Design Trending Topics (Top K Heavy Hitters)
Problem
Design a service that shows what is trending on a social platform. Users post messages containing hashtags and topics, and the service must return the top 10 trending topics for the last 5 minutes, last hour, and last 24 hours, both globally and per country. Results should be no more than a minute stale. The platform produces 500 million posts per day with about 3 topics per post, and spikes several times higher during major events.
- Does trending mean most frequent, or growing fastest relative to normal volume?
- Is an approximate ranking acceptable, or must counts be exact?
- How many countries or regions do we need separate lists for?
- Should spam and bot activity be excluded from counts?
- Do we need historical trending lists, or only the current ones?
- How are topics extracted: explicit hashtags only, or also entities from text?
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Return the top 10 topics for 5-minute, 1-hour, and 24-hour windows.
- Provide global lists and per-country lists.
- Refresh lists at least once a minute.
- Rank by velocity (current rate compared with a baseline) so perennially popular topics do not always dominate.
- Exclude topics on a policy blocklist and discount suspected bot accounts.
- Keep a history of past trending lists for analytics.
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- 500 million posts per day with 3 topics each: 1.5 billion topic events per day, about 17,000 per second on average.
- Plan for spikes of 10x, about 200,000 events per second.
- Lists at most 60 seconds stale.
- Reads are cheap and frequent: about 50,000 requests per second, served from cache.
- Approximate counts within about 1% for heavy hitters are acceptable.
- Tens of millions of distinct topics per day, most appearing only once or twice.
Topic events flow through a stream partitioned by topic. A first stage of workers counts topics in short tumbling windows per partition and emits only their local candidates, and a second stage merges those candidates into global and per-country top K lists for each window. Results are written to a cache that the read API serves directly, so reads never touch the counting pipeline.
Loading diagram...
Ingest. When a post is created, the topic extractor parses hashtags and named entities, normalizes them (lowercase, strip accents, merge aliases), drops blocklisted topics, and publishes one event per topic with topic, country, user_id, and a bot score.
Count locally. The stream is partitioned by topic hash, so all events for a topic reach the same counter worker. Each worker keeps counts in 1-minute buckets per country and globally, using exact hash maps for active topics. Every minute it emits its local top 100 per country with counts for the last 5 minutes, 1 hour, and 24 hours, computed by summing its minute buckets.
Merge globally. Because partitioning is by topic, local lists are disjoint, so the merger simply takes the union of all partitions' candidates and selects the overall top K. It divides each count by the topic's baseline rate to compute a velocity score, ranks, and writes the lists to the cache with a version timestamp.
Read. GET /trending?window=1h&country=US reads one cache key and returns the list. Clients may cache it for 30 seconds.
Turns posts into normalized topic events.
- A stateless consumer of the post stream, with a small NLP model for entity extraction and an alias table (for example mapping
#worldcup2026andworld cuptogether).
Scalability
Extraction is embarrassingly parallel and scales with post volume. Each post yields a handful of events, so this stage is rarely the bottleneck.
Tradeoffs
Normalizing and merging aliases before counting is essential; otherwise one real trend is split across variants and none of them reaches the top 10. The alias table is maintained offline, since learning it live would be complex and error prone.
Stateful stream workers that count topics in time buckets.
- Flink with keyed state in RocksDB, checkpointed to object storage for recovery.
Diagram
Loading diagram...
Data Model
- Per topic and region: a ring buffer of minute counts. Topics with no events for 24 hours are evicted.
Scalability
Partitioning by topic means no topic is counted on two workers, so merges are cheap. A single viral topic can overload one partition; to handle that, extremely hot topics are detected and split with a salt suffix across several workers, then summed in the merger.
Tradeoffs
Keeping per-minute buckets only for topics active in the last day bounds memory to tens of millions of small ring buffers. For the long tail, a Count-Min Sketch plus a heap of heavy hitters could replace exact maps, trading a small overcount error for fixed memory; it becomes worthwhile if distinct topics grow by another order of magnitude.
Combines partial results into final lists and applies ranking.
- A single Flink operator (with a standby) or a small service consuming the partial top K stream.
Scalability
Input is at most a few hundred partitions times 100 candidates times the number of regions per minute, which is small. One instance handles it; the standby takes over on failure.
Tradeoffs
Ranking by velocity (current count divided by a smoothed baseline for that topic and hour of day) surfaces new spikes rather than evergreen topics like #music. Pure frequency ranking is simpler but makes the list nearly static.
Serves the latest lists to clients.
- Redis with one key per window and region, plus CDN caching for logged-out reads.
API
GET /trending?window=5m|1h|24h&country=returns{ topics: [{ topic, score, postCount }], generatedAt }.
Data Model
- Key
trending:{window}:{country}holding a serialized list andgenerated_at.
Scalability
There are only a few hundred keys, each read thousands of times per second, so replicas and client caching handle load easily.
Tradeoffs
Precomputing every list each minute decouples read traffic from computation entirely. Computing on demand would let users pick arbitrary windows but would put heavy queries on the hot read path.
Reduces the influence of bots and coordinated campaigns on counts.
- Bot scores from the trust and safety system, attached to each event at extraction time.
Scalability
The filter is just a weight applied per event, so it costs nothing extra in the counters. Counting distinct users per topic with a HyperLogLog per bucket further limits the impact of a few accounts posting repeatedly.
Tradeoffs
Weighting by distinct users rather than raw posts makes trends harder to manipulate but slightly delays detection of genuine spikes driven by a few prolific accounts, such as live event commentators. Product teams accept that delay.