Skip to main content
PreparationZone Logo
PrepZone

Broker Sizing, Partitions, and JVM Tuning

Disk, network, heap, and partition count guidelines for a VaultCommerce production cluster.

Pass checkpoint below

Why this matters

  • VaultCommerce production: 6 brokers, 2TB NVMe each, 12 partitions per hot topic.
  • Kafka is I/O bound — heap 6GB, G1GC; rely on OS cache for page reads.
  • Network: 10 Gbps NIC; replication traffic doubles write volume with RF=3.
  • Under-sized clusters show up as disk full and fetch latency long before CPU maxes.
ProducersOrder, payment services
Broker cluster3+ brokers, KRaft
Consumer groupsInventory, analytics
Topic: vaultcommerce.orders.placed.v1
Partition 0Leader on broker-1
Partition 1Leader on broker-2
Partition 2Leader on broker-3
Producers write to topic partitions on brokers. Consumers read via consumer groups.

Disk and retention math

Daily ingress × retention days × RF = raw disk. VaultCommerce: 500GB/day × 7 × 3 = 10.5TB cluster raw; spread across 6 brokers ≈ 1.75TB/broker + 30% headroom. Monitor kafka.log:type=Log,name=Size.

Key points

  • Retention sizing — ingress rate × days × replication factor
  • Page cache — Kafka relies on OS cache, not JVM heap for hot data
  • Network bandwidth — producer ingress + replication + consumer fetch
  • Partition count — target < 4000 partitions per broker for metadata health
  • num.io.threads / num.network.threads — broker request parallelism

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • JVM and OS tuning
  • VaultCommerce rollout checklist

Test yourself

Answer these before moving on — recall is what makes it stick.