Learning track 06
Kafka & messaging, fresher to expert
Ten modules from async messaging fundamentals to production Kafka operations. VaultCommerce's event backbone grows with every chapter — one platform, zero gaps.
- Articles
- 52
- Modules
- 10
- Total read
- 9 hr 32 min
- Streak
- 0 days
Start here
Why Systems Need Async Messaging
01
Messaging Foundations
Why async messaging exists, delivery semantics, and the queue landscape
0/5
- Why Systems Need Async MessagingDecouple services, absorb spikes, and survive partial failures — why VaultCommerce moved off synchronous HTTP fan-out.
- Sync vs Async: Latency, Coupling, and FailureWhen to call synchronously and when to publish an event — trade-offs every backend engineer must articulate.
- Queue, Log, and Pub-Sub LandscapeRabbitMQ, SQS, Redis Streams, Pulsar, and Kafka — what each optimizes for and where VaultCommerce fits.
- At-Most, At-Least, and Exactly-Once DeliveryDelivery guarantees explained with VaultCommerce order events — what brokers promise and what your code must enforce.
- Ordering Guarantees and Idempotent HandlersWhy duplicate messages happen and how idempotent consumers keep VaultCommerce inventory correct.
02
Kafka Core Architecture
Brokers, topics, partitions, replication, KRaft, and log storage
0/6
- What Apache Kafka Is (and Is Not)A distributed commit log for high-throughput event streaming — not a task queue, not a database.
- Brokers, Topics, and PartitionsHow VaultCommerce order events land on brokers, topics, and partitions — the physical layout of a Kafka cluster.
- Replication, ISR, Leaders, and FollowersHow Kafka survives broker loss without losing committed messages — leaders, followers, and the in-sync replica set.
- KRaft: Kafka Without ZooKeeperModern Kafka runs metadata in KRaft quorum — faster failover, simpler ops, and the only path forward on Kafka 4.x.
- Log Segments, Offsets, and Disk LayoutSegments, offsets, and retention on disk — how Kafka stores billions of VaultCommerce events efficiently.
- Kafka vs RabbitMQ, SQS, and Redis StreamsDeep comparison when VaultCommerce picks a log over a queue — replay, throughput, and operational trade-offs.
03
Producers
Java client, send API, partitioning, acks, batching, and idempotence
0/6
- Java Producer Client SetupBootstrap servers, serializers, and your first VaultCommerce order event published with the Apache Kafka Java client.
- Send API: Sync, Async, and CallbacksRecordMetadata, CompletableFuture callbacks, and when VaultCommerce blocks on publish vs fires-and-forgets.
- Record Keys and Partition SelectionPartition by orderId so all events for one VaultCommerce checkout stay ordered on a single partition.
- acks, Retries, and Durability Tuningacks=all, min.insync.replicas, and retry policy — the producer settings that prevent silent data loss.
- Batching, linger.ms, and CompressionTrade a few milliseconds of latency for 10x throughput — batching and compression on the VaultCommerce producer.
- Idempotent Producer and Sequence Numbersenable.idempotence=true prevents duplicate writes on retry — essential before VaultCommerce enables transactions.
04
Consumers
Consumer groups, offsets, rebalancing, poll loop, and lag
0/5
- Consumer Groups and Partition AssignmentOne consumer per partition per group — how VaultCommerce inventory and payment services scale independently.
- Committed Offsets and auto.offset.resetWhere each VaultCommerce consumer group resumes after restart — committed offsets and reset policies.
- Rebalance: Cooperative Sticky vs RangeWhat happens when consumers join or leave — cooperative sticky rebalancing minimizes duplicate processing.
- Poll Loop, max.poll.interval, and HeartbeatsWhy long-running VaultCommerce handlers must not block poll() — heartbeats, session timeout, and max.poll.interval.ms.
- Consumer Lag, Scaling, and BackpressureLag as the primary health signal — scale consumers up to partition count and apply backpressure when saturated.
05
Topics, Schemas, and Security
Topic design, compaction, schema registry, serialization, and ACLs
0/5
- Topic Design, Naming, and Partition Countvaultcommerce.orders.placed.v1 — naming conventions, partition count, and replication factor for production topics.
- Retention, Compaction, and TombstonesTime-based retention vs log compaction for VaultCommerce customer profile changelog topics.
- Schema Registry and Compatibility ModesAvro schemas, BACKWARD compatibility, and evolving OrderPlaced v1 to v2 without breaking consumers.
- Avro, Protobuf, and JSON Trade-offsPick serialization for VaultCommerce events — schema evolution, payload size, and human debuggability.
- ACLs, SASL, and TLS BasicsPrinciple of least privilege on VaultCommerce topics — SASL/SCRAM, TLS, and ACL patterns.
06
Integration and CDC
Kafka Connect, Debezium, mirroring, and event sourcing
0/4
- Kafka Connect: Sources, Sinks, and SMTsMove data in and out of Kafka without custom consumers — JDBC source, S3 sink, and single message transforms.
- Debezium CDC from VaultCommerce PostgresCapture Postgres WAL changes and stream them to Kafka — the bridge between OLTP and search indexes.
- MirrorMaker 2 and Multi-Region ReplicationReplicate VaultCommerce topics across regions for disaster recovery and read-local consumers.
- Event Sourcing on a Kafka LogRebuild VaultCommerce order state from the event log — snapshots, replay, and temporal queries.
07
Stream Processing
Kafka Streams, windowing, state stores, ksqlDB, and Flink
0/5
- Stream vs Batch Processing Mental ModelProcess VaultCommerce clickstreams as unbounded data — event time, processing time, and watermarks.
- Kafka Streams: Topology and DSLBuild a VaultCommerce fraud-detection topology with map, filter, groupBy, and aggregate operators.
- Tumbling, Hopping, and Session WindowsCount VaultCommerce orders per five-minute window — tumbling, hopping, and session window semantics.
- State Stores, RocksDB, and Changelog TopicsFault-tolerant local state in Kafka Streams — RocksDB on disk backed by compacted changelog topics.
- ksqlDB and Apache Flink OverviewWhen VaultCommerce graduates from Kafka Streams to ksqlDB or Flink for complex joins and large state.
08
Reliability Patterns
Transactions, DLT, outbox, sagas, and ordering strategy
0/5
- Kafka Transactions and EOS SemanticsAtomic write across multiple partitions — read-process-write with transactional.id and isolation levels.
- DLT, Retry Topics, and Poison MessagesRoute poison VaultCommerce payloads to a dead-letter topic after bounded retries — never block the partition.
- Transactional Outbox Pattern (Deep Dive)Atomically save the order and the outbox row — then relay to Kafka without dual-write races.
- Saga Choreography with Domain EventsOrderPlaced → ReserveInventory → ChargePayment → ShipOrder — compensating events on failure.
- Ordering, Causality, and Partition StrategyDesign partition keys so causally related VaultCommerce events always land on the same partition.
09
Java Ecosystem
Spring Kafka, error handling, Testcontainers, and local dev
0/4
- Spring Kafka: Template, Listeners, and ConfigKafkaTemplate and @KafkaListener for VaultCommerce — Spring Boot 3 auto-configuration and JSON serialization.
- Error Handlers, Retry, and DLT in SpringDefaultErrorHandler, exponential backoff, and DeadLetterPublishingRecoverer for VaultCommerce consumers.
- Integration Testing with TestcontainersSpin up a real Kafka broker in JUnit — publish and consume VaultCommerce events in CI without mocks.
- Local Kafka Cluster with Docker ComposeKRaft-mode three-broker compose stack for VaultCommerce local development with Schema Registry.
10
Production and Expert Playbook
Sizing, security, monitoring, upgrades, troubleshooting, and on-call
0/7
- Broker Sizing, Partitions, and JVM TuningDisk, network, heap, and partition count guidelines for a VaultCommerce production cluster.
- Production Security ChecklistmTLS, SCRAM-SHA-512, ACL audit, and secret rotation — the VaultCommerce Kafka security baseline.
- Metrics, JMX, Prometheus, and Lag AlertsUnder-replicated partitions, offline replicas, and consumer lag alerts that wake the on-call engineer.
- Rolling Upgrades and KRaft MigrationUpgrade Kafka 3.7 → 4.x with zero downtime — rolling broker restarts and KRaft migration checklist.
- Troubleshooting: Lag, Rebalance, DiskStep-by-step diagnosis for the five Kafka incidents VaultCommerce sees most often in production.
- Interview Reference SheetDense reference card — architecture, semantics, tuning knobs, and trade-offs for senior backend interviews.
- On-Call Runbook and Command Cheat SheetThe commands and checks every VaultCommerce on-call engineer runs before escalating a Kafka incident.