Skip to main content
PreparationZone Logo
PrepZone

Message Queues Fundamentals

Decouple producers and consumers with durable queues, acknowledgements and back-pressure.

Pass checkpoint below

Read these first

When StreamHub finishes transcoding a VOD upload, the API cannot synchronously notify every follower, update search indexes, and send push alerts in one HTTP response. It publishes a video.ready event to a queue and returns immediately while workers handle the rest.

Why queues exist

Problems queues solve

  • Decoupling: Producer and consumer scale, deploy, and fail independently.
  • Spike absorption: A flash sale or viral stream queues work instead of overwhelming downstream services.
  • Reliability: Messages persist on disk until acknowledged — survive consumer crashes.
  • Async workflows: One event triggers multiple consumers (notifications, analytics, search indexing).

Amazon MSK event pipeline

produceCOMPUTE
EKS APIorder placed
INTEGRATION
Amazon MSKorders.placed.v1
COMPUTE
InventoryEKS worker
COMPUTE
Email svcEKS worker
ANALYTICS
AnalyticsFlink / EMR
API publishes events; worker fleets scale independently on consumer lag.

Core concepts

ConceptMeaningStreamHub example
ProducerService that publishes messagesUpload service after transcode completes
ConsumerService that reads and processes messagesNotification worker, search indexer
Queue / TopicDurable buffer between themSQS queue or Kafka topic `video.events`
AcknowledgementConsumer confirms successful processingDelete from SQS or commit Kafka offset
Dead letter queueHolds messages that failed max retriesMalformed transcode events for manual review
  • Producer

    MeaningService that publishes messages
    StreamHub exampleUpload service after transcode completes
  • Consumer

    MeaningService that reads and processes messages
    StreamHub exampleNotification worker, search indexer
  • Queue / Topic

    MeaningDurable buffer between them
    StreamHub exampleSQS queue or Kafka topic `video.events`
  • Acknowledgement

    MeaningConsumer confirms successful processing
    StreamHub exampleDelete from SQS or commit Kafka offset
  • Dead letter queue

    MeaningHolds messages that failed max retries
    StreamHub exampleMalformed transcode events for manual review

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 4 sections

  • Delivery guarantees
  • Message anatomy
  • Back-pressure and flow control
  • StreamHub upload pipeline

Test yourself

Answer these before moving on — recall is what makes it stick.