RabbitMQ and Kafka Patterns

Point-to-point versus log-based messaging — when to pick each and how to configure them.

AdvancedTheory13 min read

Read these first

Message Queues Fundamentals

StreamHub uses RabbitMQ for task queues (transcode jobs with priority routing) and Kafka for event streams (viewer analytics, audit logs, and cross-service fan-out). Picking the wrong one costs operability or replay capability.

Mental model difference

Aspect	RabbitMQ (smart broker)	Kafka (dumb broker, smart consumers)
Data model	Messages deleted after ack	Append-only log retained by time/size
Delivery	Broker pushes to consumers	Consumers pull at their offset
Routing	Exchanges, bindings, routing keys	Topics + partitions; no built-in routing
Replay	Not designed for replay	Reset offset and re-read history
Throughput	Thousands/sec per queue	Millions/sec per cluster
Best for	Task queues, RPC, priority jobs	Event streaming, analytics, audit

Data model
RabbitMQ (smart broker)Messages deleted after ack
Kafka (dumb broker, smart consumers)Append-only log retained by time/size
Delivery
RabbitMQ (smart broker)Broker pushes to consumers
Kafka (dumb broker, smart consumers)Consumers pull at their offset
Routing
RabbitMQ (smart broker)Exchanges, bindings, routing keys
Kafka (dumb broker, smart consumers)Topics + partitions; no built-in routing
Replay
RabbitMQ (smart broker)Not designed for replay
Kafka (dumb broker, smart consumers)Reset offset and re-read history
Throughput
RabbitMQ (smart broker)Thousands/sec per queue
Kafka (dumb broker, smart consumers)Millions/sec per cluster
Best for
RabbitMQ (smart broker)Task queues, RPC, priority jobs
Kafka (dumb broker, smart consumers)Event streaming, analytics, audit

Amazon MSK event pipeline

API publishes events; worker fleets scale independently on consumer lag.

RabbitMQ patterns

Exchange types

Direct: Route by exact routing key — transcode.hd → HD worker queue.
Fanout: Broadcast to all bound queues — deploy notifications to every service.
Topic: Pattern match on routing key — video.*.ready matches video.hd.ready.
Headers: Route by message header attributes — multi-tenant routing by tenant_id.

Java

# RabbitMQ topology (pseudo-config)
exchanges:
  - name: streamhub.tasks
    type: topic
queues:
  - name: transcode.hd
    bindings:
      - exchange: streamhub.tasks
        routing_key: transcode.hd
  - name: transcode.sd
    bindings:
      - exchange: streamhub.tasks
        routing_key: transcode.sd

Kafka patterns

Kafka building blocks

Topic: Named log split into ordered partitions.
Partition: Unit of parallelism — one consumer per partition in a group.
Consumer group: Competing consumers share partition assignment.
Offset: Consumer's read position — commit after processing for at-least-once.
Retention: Messages kept 7 days (configurable) regardless of consumption.

Java

# Kafka topic config (pseudo)
topics:
  - name: viewer.events
    partitions: 24
    replication_factor: 3
    retention_ms: 604800000   # 7 days
    cleanup_policy: delete

Partition key = streamer_id keeps all events for one streamer ordered on one partition.

When to pick which

Use case	Pick	Why
Transcode job queue with priorities	RabbitMQ	Smart routing, per-message ack, DLQ built-in
Viewer analytics event stream	Kafka	High throughput, replay for batch jobs
Payment event bus	Kafka	Audit trail, multiple consumers, retention
Delayed retry with TTL	RabbitMQ	Per-message TTL and dead-letter exchanges
Cross-region event replication	Kafka MirrorMaker	Log-based replication at scale

Transcode job queue with priorities
PickRabbitMQ
WhySmart routing, per-message ack, DLQ built-in
Viewer analytics event stream
PickKafka
WhyHigh throughput, replay for batch jobs
Payment event bus
PickKafka
WhyAudit trail, multiple consumers, retention
Delayed retry with TTL
PickRabbitMQ
WhyPer-message TTL and dead-letter exchanges
Cross-region event replication
PickKafka MirrorMaker
WhyLog-based replication at scale

StreamHub hybrid architecture

Cloud building blocks toolbox

NETWORKRoute 53DNS + GeoDNS

NETWORKCloudFrontCDN edge

NETWORKAWS ALBL7 balance

NETWORKAPI Gatewayauth + throttle

COMPUTEAmazon EKScontainer fleet

DATABASEElastiCacheRedis cache

DATABASERDS / Aurorarelational

DATABASEDynamoDBkey-value

INTEGRATIONMSK / SQSasync events

STORAGEAmazon S3object store

ANALYTICSOpenSearchfull-text

OPSCloudWatchmetrics · logs

SECURITYAWS WAFedge security

SECURITYCognitoidentity

COMPUTELambdaserverless

INTEGRATIONSNS / SESnotify

Every StreamHub product design combines subsets of these AWS components.

API writes to RabbitMQ for immediate task dispatch; workers publish results to Kafka for downstream fan-out. Analytics team consumes Kafka independently without touching the task queue.

Kafka trackGo deeper: Kafka internals, client configs, and VaultCommerce production patterns

Quick recall

Everything you need if you only revisit this box.

RabbitMQ: smart broker, push delivery, messages deleted after ack — great for task queues.
Kafka: append-only log, pull by offset, retained for replay — great for event streams.
Kafka partitions provide ordering within a partition; partition key controls co-location.
Consumer groups in Kafka enable competing consumers and independent replay groups.
StreamHub uses RabbitMQ for jobs, Kafka for events — hybrid is common at scale.
Always design Kafka consumers to be idempotent; offsets commit after processing.

Test yourself

Answer these before moving on — recall is what makes it stick.