Skip to main content
PreparationZone Logo
PrepZone

Distributed Message Queue

Build a queue service with partitions, consumer groups and at-least-once delivery.

Pass checkpoint below

Read these first

Design a queue service for StreamHub's internal teams: publish messages to named topics, consume in consumer groups with offset tracking, retain messages for replay, and scale to 1M messages/sec.

Requirements

Functional requirements

  • produce(topic, message) — append message to topic partition.
  • consume(topic, group, offset) — read messages from assigned partitions.
  • Consumer groups: competing consumers share partition assignment.
  • Message retention by time (7 days) and size (1 TB per topic).
  • Dead letter topic for messages exceeding max retry count.

Non-functional requirements

  • Throughput: 1M messages/sec aggregate across topics.
  • Latency: p99 produce under 10 ms; p99 consume under 5 ms (after publish).
  • Durability: Replicate each message to 3 brokers before ack.
  • Availability: Tolerate broker failure without message loss.
  • Ordering: Per-partition ordering guaranteed; cross-partition unordered.

Estimation

  • 1M msg/sec × 1 KB avg = 1 GB/sec ingest → 86 TB/day raw → replication factor 3 = 258 TB/day.
  • 100 topics × 24 partitions × 3 replicas = 7200 partition-replicas — ~50 broker cluster.

Amazon MSK event pipeline

produceCOMPUTE
EKS APIorder placed
INTEGRATION
Amazon MSKorders.placed.v1
COMPUTE
InventoryEKS worker
COMPUTE
Email svcEKS worker
ANALYTICS
AnalyticsFlink / EMR
API publishes events; worker fleets scale independently on consumer lag.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 4 sections

  • API design
  • Topic and partition design
  • Storage architecture
  • Consumer groups

Test yourself

Answer these before moving on — recall is what makes it stick.