Design a Distributed Message Queue
Problem
Design a message queue that lets producer services publish messages to named topics and lets consumer services read them asynchronously. Messages within a partition must be delivered in the order they were written, messages must not be lost once the broker acknowledges them, and multiple independent consumer groups must be able to read the same topic, each at its own pace. The system must sustain 2 million messages per second at peak with an average message size of 1 KB, and messages are retained for 7 days so consumers can replay history.
- Is ordering required across a whole topic, or only for messages that share a key?
- Which delivery guarantee do consumers need: at most once, at least once, or exactly once?
- Should messages be deleted when consumed, or kept for a retention period?
- What end-to-end latency is acceptable from publish to consume?
- Do we need delayed messages, priorities, or per-message acknowledgment like a task queue?
- Is the cluster deployed in one region or several?
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Producers publish messages to a named topic, optionally with a key.
- Consumers in a group read each message once per group, in order within a partition.
- Multiple consumer groups read the same topic independently.
- Retain messages for a configurable period (default 7 days) and allow replay from an earlier offset.
- Create, delete, and change the partition count of topics.
- Route messages that repeatedly fail processing to a dead letter topic.
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Sustain 2 million messages per second at peak, or about 2 GB per second of incoming data.
- No loss of acknowledged messages, even if a broker fails.
- Store 7 days of data: 2 GB per second at an average of half the peak is about 600 TB, or 1.8 PB with three replicas.
- Publish to consume latency under 100 ms at p99 in normal operation.
- High availability: a single broker failure causes at most a few seconds of unavailability for its partitions.
- Horizontal scalability by adding brokers and partitions.
Each topic is split into partitions, and each partition is an append-only log replicated across three brokers with one leader. Producers send batches to partition leaders, which acknowledge once the in-sync replicas have the data; consumers pull batches and track their position as an offset per partition. A replicated controller assigns partitions to brokers, elects new leaders on failure, and coordinates consumer group membership.
Loading diagram...
Publish. The producer client fetches metadata to learn which broker leads each partition. It picks a partition by hashing the message key (or round robin when there is no key), accumulates messages into a batch for a few milliseconds, compresses it, and sends it to the leader. The leader appends the batch to its log segment. Followers fetch from the leader continuously, and once every in-sync replica has the batch the leader advances the high watermark and acknowledges the producer.
Consume. Consumers in a group join through a group coordinator, which assigns each partition to exactly one consumer in the group. Each consumer repeatedly asks the partition leader for messages after its current offset, processes them, and commits the new offset. Consumers only see messages below the high watermark, so they never read data that could be lost in a failover.
Broker failure. The controller detects the missed heartbeat, chooses a new leader from the partition's in-sync replica set, and publishes updated metadata. Producers and consumers refresh metadata and continue with the new leader.
Brokers store partitions as append-only logs and serve reads and writes for the partitions they lead.
- JVM or Rust service writing to local SSD or NVMe, relying on the OS page cache.
- Zero-copy transfer (
sendfile) from page cache to the network for consumers.
Diagram
Loading diagram...
Data Model
- Log record:
offset,timestamp,key,value,headers, grouped into compressed batches with a CRC. - Segments of about 1 GB each, with a sparse index from offset to file position.
Scalability
Sequential appends and sequential reads let each broker sustain hundreds of MB per second. With 2 GB per second of input and three replicas, about 6 GB per second of disk writes are spread over roughly 40 brokers. Older segments are moved to object storage, so brokers keep only recent data on local disk while the full 7 days stays readable.
Tradeoffs
Keeping messages in a log after consumption, instead of deleting them on acknowledgment like a traditional queue, enables replay and many consumer groups at no extra cost. The price is that per-message acknowledgment and priorities are not natural; workloads that need them are better served by a task queue layered on top or a different product.
Copies every partition to multiple brokers so an acknowledged message survives failures.
- Leader and follower replication with an in-sync replica (ISR) set per partition.
Data Model
- Per partition:
leader,replicas[],isr[],leaderEpoch,highWatermark.
Scalability
Followers pull in large batches, so replication adds little overhead per message. Leaders for different partitions are spread evenly across brokers, so every broker shares the leader load.
Tradeoffs
With replication factor 3 and min.insync.replicas = 2, a write with acks=all survives one broker failure while staying available. Waiting for all three replicas would block writes whenever any broker is slow. A majority quorum protocol like Raft per partition gives similar guarantees, but ISR replication tolerates more failures for the same replica count. Leader epochs prevent a stale leader from accepting writes after a new one is elected.
Library that batches, partitions, and retries publishes.
API
send(topic, key?, value, headers?) -> future of { partition, offset }- Settings:
acks(0, 1, or all),linger.ms,batch.size,compression,enable.idempotence.
Scalability
Batching for 5 to 10 ms and compressing batches cuts the number of requests and the bytes on the wire by an order of magnitude, which is how the cluster reaches millions of messages per second.
Tradeoffs
Retries can create duplicates if an acknowledgment is lost. An idempotent producer attaches a producer ID and sequence number to each batch so the broker can discard duplicates. Larger linger times raise throughput but add latency, so the default is small and tunable per producer.
Divides partitions among consumers and records how far each group has read.
- A group coordinator on the brokers, with offsets stored in an internal compacted topic.
API
subscribe(topics, groupId),poll(timeout) -> records,commit(offsets),seek(partition, offset).
Data Model
- Committed offsets: key
(groupId, topic, partition), value{ offset, metadata, commitTime }.
Scalability
Parallelism within a group is capped at the partition count, so topics are created with headroom (for example 64 or more partitions for a busy topic). Cooperative rebalancing moves only the partitions that need to change hands when consumers join or leave, instead of pausing the whole group.
Tradeoffs
Committing offsets after processing gives at-least-once delivery, so consumers must be idempotent. Committing before processing gives at-most-once and risks loss. Exactly-once is possible within the system through transactions that write outputs and offsets atomically, but it adds latency and only covers outputs written back to the queue, so at-least-once with idempotent consumers is the default.
Manages cluster metadata: topics, partition assignments, leaders, and broker membership.
- A Raft quorum of three or five controller nodes storing metadata as a replicated log, similar to Kafka KRaft.
Scalability
Brokers cache metadata and receive incremental updates, so the controller's load depends on metadata changes, not message traffic. Failover of thousands of partition leaders completes in seconds because elections are made centrally from the ISR, not negotiated per partition.
Tradeoffs
An external coordinator like ZooKeeper was the traditional choice, but it is another system to operate and limits how many partitions the cluster can manage. Building Raft into the controllers removes that dependency and speeds up failover.