Why this matters
- VaultCommerce's 12-partition order topic allows 12 inventory consumers to process in parallel while preserving per-order ordering.
- Partition count is the upper bound on consumer parallelism within a group.
- Replication factor determines fault tolerance — RF=3 survives two broker losses with min.insync.replicas=2.
- Physical layout knowledge prevents misconfigured topics in production.
Broker responsibilities
Each broker hosts partition leaders and followers, serves produce/fetch requests, and participates in KRaft metadata quorum (on combined or dedicated controller nodes). VaultCommerce runs 6 brokers across 3 AZs — 3 controllers, 6 data nodes in combined mode for cost efficiency at moderate scale.
Key points
- Broker — Kafka server process storing and serving partition data
- Leader partition — broker that handles all reads and writes for that partition replica
- Partition — ordered, immutable sequence of records
- Replication factor — number of copies across brokers (VaultCommerce production: 3)
- Topic naming —
vaultcommerce.{domain}.{event}.v{version}convention
Choosing partition count
Start with peak throughput ÷ per-partition throughput. VaultCommerce measured ~4k msg/s per partition with 512-byte Avro records and acks=all. Target 50k peak → 12 partitions with headroom. Increasing partitions later does not redistribute existing keys — plan upfront.
VaultCommerce rollout checklist
Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. Partitions enable parallelism; keys enable per-entity ordering. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.
# Create VaultCommerce order topic — 12 partitions, RF=3, min ISR enforced at cluster level
kafka-topics.sh --bootstrap-server kafka-1:9092 --create \
--topic vaultcommerce.orders.placed.v1 \
--partitions 12 \
--replication-factor 3 \
--config min.insync.replicas=2 \
--config retention.ms=604800000
Quick recall
Everything you need if you only revisit this box.
- Partitions enable parallelism; keys enable per-entity ordering.
- Consumer count > partition count wastes resources.
- RF=3 and min.insync.replicas=2 prevent silent loss on acks=all.
Test yourself
Answer these before moving on — recall is what makes it stick.