Design a queue service for StreamHub's internal teams: publish messages to named topics, consume in consumer groups with offset tracking, retain messages for replay, and scale to 1M messages/sec.
Requirements
Functional requirements
produce(topic, message)— append message to topic partition.consume(topic, group, offset)— read messages from assigned partitions.- Consumer groups: competing consumers share partition assignment.
- Message retention by time (7 days) and size (1 TB per topic).
- Dead letter topic for messages exceeding max retry count.
Non-functional requirements
- Throughput: 1M messages/sec aggregate across topics.
- Latency: p99 produce under 10 ms; p99 consume under 5 ms (after publish).
- Durability: Replicate each message to 3 brokers before ack.
- Availability: Tolerate broker failure without message loss.
- Ordering: Per-partition ordering guaranteed; cross-partition unordered.
Estimation
- 1M msg/sec × 1 KB avg = 1 GB/sec ingest → 86 TB/day raw → replication factor 3 = 258 TB/day.
- 100 topics × 24 partitions × 3 replicas = 7200 partition-replicas — ~50 broker cluster.
Amazon MSK event pipeline
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 4 sections
- API design
- Topic and partition design
- Storage architecture
- Consumer groups
Test yourself
Answer these before moving on — recall is what makes it stick.