Local Kafka Cluster with Docker Compose

KRaft-mode three-broker compose stack for VaultCommerce local development with Schema Registry.

Intermediate9 min read

Read these first

Integration Testing with Testcontainers

Why this matters

Every engineer runs docker compose up kafka — identical topic creation scripts as staging.
KRaft 3-broker compose matches production metadata behavior at laptop scale.
Schema Registry + AKHQ for debugging Avro payloads.
Faster iteration than shared dev cluster contention.

ProducersOrder, payment services

Broker cluster3+ brokers, KRaft

Consumer groupsInventory, analytics

Topic: vaultcommerce.orders.placed.v1

Partition 0Leader on broker-1

Partition 1Leader on broker-2

Partition 2Leader on broker-3

Producers write to topic partitions on brokers. Consumers read via consumer groups.

Compose topology

3 Kafka brokers in KRaft combined mode, 1 Schema Registry, 1 AKHQ. Host ports 9092/9093/9094 for bootstrap. VaultCommerce docker/compose/kafka.yml seeds topics via kafka-init one-shot container.

Key points

KRaft combined mode — broker + controller in one container for dev
Schema Registry — local Avro schema registration on push
kafka-init — container that creates topics after brokers healthy
AKHQ / Kafka UI — browse topics, consumer groups, messages
advertised.listeners — PLAINTEXT://localhost:9092 for host access

Application wiring

application-local.yml points to localhost:9092. SCRAM disabled locally; ACLs off. Producers use same topic names with .local suffix optional — VaultCommerce uses identical names, isolated by local cluster.

VaultCommerce rollout checklist

Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. 3-broker KRaft compose mirrors prod metadata. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.

Java

# VaultCommerce docker/compose/kafka.yml excerpt — KRaft 3.7
services:
  kafka-1:
    image: apache/kafka:3.7.0
    hostname: kafka-1
    ports: ["9092:9092"]
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      CLUSTER_ID: "vaultcommerce-local-kraft-01"

  schema-registry:
    image: confluentinc/cp-schema-registry:7.7.0
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka-1:9092

Quick recall

Everything you need if you only revisit this box.

3-broker KRaft compose mirrors prod metadata.
Schema Registry for Avro local dev.
advertised.listeners must be reachable from host.

Test yourself

Answer these before moving on — recall is what makes it stick.