When StreamHub finishes transcoding a VOD upload, the API cannot synchronously notify every follower, update search indexes, and send push alerts in one HTTP response. It publishes a video.ready event to a queue and returns immediately while workers handle the rest.
Why queues exist
Problems queues solve
- Decoupling: Producer and consumer scale, deploy, and fail independently.
- Spike absorption: A flash sale or viral stream queues work instead of overwhelming downstream services.
- Reliability: Messages persist on disk until acknowledged — survive consumer crashes.
- Async workflows: One event triggers multiple consumers (notifications, analytics, search indexing).
Amazon MSK event pipeline
Core concepts
| Concept | Meaning | StreamHub example |
|---|---|---|
| Producer | Service that publishes messages | Upload service after transcode completes |
| Consumer | Service that reads and processes messages | Notification worker, search indexer |
| Queue / Topic | Durable buffer between them | SQS queue or Kafka topic `video.events` |
| Acknowledgement | Consumer confirms successful processing | Delete from SQS or commit Kafka offset |
| Dead letter queue | Holds messages that failed max retries | Malformed transcode events for manual review |
Producer
MeaningService that publishes messagesStreamHub exampleUpload service after transcode completesConsumer
MeaningService that reads and processes messagesStreamHub exampleNotification worker, search indexerQueue / Topic
MeaningDurable buffer between themStreamHub exampleSQS queue or Kafka topic `video.events`Acknowledgement
MeaningConsumer confirms successful processingStreamHub exampleDelete from SQS or commit Kafka offsetDead letter queue
MeaningHolds messages that failed max retriesStreamHub exampleMalformed transcode events for manual review
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 4 sections
- Delivery guarantees
- Message anatomy
- Back-pressure and flow control
- StreamHub upload pipeline
Test yourself
Answer these before moving on — recall is what makes it stick.