Why this matters
- VaultCommerce's 7-day order retention means ~2 TB per broker — segment sizing affects delete performance.
- Sequential disk writes are why Kafka outperforms random-write databases for ingestion.
- Understanding segments explains offset gaps, log compaction, and disk full incidents.
- Ops teams use segment metrics to plan broker disk upgrades before Black Friday.
Segment files and indexes
Active segment receives new writes (000000000123456789.log). When segment.bytes or segment.ms is hit, the segment rolls. Sparse offset and time indexes enable O(1) offset-to-position lookup. VaultCommerce sets log.segment.bytes=1GB and log.segment.ms=604800000 (7 days max per segment).
Key points
- Segment — immutable file chunk of a partition log; active segment alone accepts writes
- Log end offset (LEO) — next offset to be written on the leader
- High watermark (HW) — offset replicated to all ISR; consumers see up to HW-1
- retention.ms — time-based segment deletion policy
- log.dirs — comma-separated data directories; VaultCommerce uses 3 NVMe mounts per broker
Retention and disk planning
retention.ms deletes segments whose last record timestamp exceeds the window. Compacted topics use min.cleanable.dirty.ratio and tombstones instead. VaultCommerce monitors Log_Size per topic and alerts at 70% disk — expanding EBS before segment delete can catch up.
VaultCommerce rollout checklist
Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. Partitions are append-only segment files on disk. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.
# VaultCommerce ops — inspect segment layout for hot partition
ls -lh /var/kafka/data/vaultcommerce.orders.placed.v1-0/
# 00000000000000000000.log 00000000000000000000.index 00000000000000000000.timeindex
# 000000000123456789.log ...
kafka-log-dirs.sh --bootstrap-server kafka-1:9092 \
--describe --topic-list vaultcommerce.orders.placed.v1 \
--json | jq '.brokers[].logDirs[].partitions[] | select(.size > 100000000000)'
Quick recall
Everything you need if you only revisit this box.
- Partitions are append-only segment files on disk.
- Retention deletes segments, not individual messages.
- Monitor disk per broker before retention lag causes outages.
Test yourself
Answer these before moving on — recall is what makes it stick.