Why this matters
- VaultCommerce production: 6 brokers, 2TB NVMe each, 12 partitions per hot topic.
- Kafka is I/O bound — heap 6GB, G1GC; rely on OS cache for page reads.
- Network: 10 Gbps NIC; replication traffic doubles write volume with RF=3.
- Under-sized clusters show up as disk full and fetch latency long before CPU maxes.
ProducersOrder, payment services
Broker cluster3+ brokers, KRaft
Consumer groupsInventory, analytics
Topic: vaultcommerce.orders.placed.v1
Partition 0Leader on broker-1
Partition 1Leader on broker-2
Partition 2Leader on broker-3
Disk and retention math
Daily ingress × retention days × RF = raw disk. VaultCommerce: 500GB/day × 7 × 3 = 10.5TB cluster raw; spread across 6 brokers ≈ 1.75TB/broker + 30% headroom. Monitor kafka.log:type=Log,name=Size.
Key points
- Retention sizing — ingress rate × days × replication factor
- Page cache — Kafka relies on OS cache, not JVM heap for hot data
- Network bandwidth — producer ingress + replication + consumer fetch
- Partition count — target < 4000 partitions per broker for metadata health
- num.io.threads / num.network.threads — broker request parallelism
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- JVM and OS tuning
- VaultCommerce rollout checklist
Test yourself
Answer these before moving on — recall is what makes it stick.