Skip to main content
PreparationZone Logo
PrepZone

Log Segments, Offsets, and Disk Layout

Segments, offsets, and retention on disk — how Kafka stores billions of VaultCommerce events efficiently.

Pass checkpoint below

Why this matters

  • VaultCommerce's 7-day order retention means ~2 TB per broker — segment sizing affects delete performance.
  • Sequential disk writes are why Kafka outperforms random-write databases for ingestion.
  • Understanding segments explains offset gaps, log compaction, and disk full incidents.
  • Ops teams use segment metrics to plan broker disk upgrades before Black Friday.
Partition log
Segment 0offsets 0–999
Segment 1offsets 1000–1999
Active segmentoffsets 2000+
Consumer offset 1850Points into segment 1
Each partition is an append-only sequence of segments. Consumers track position by offset.

Segment files and indexes

Active segment receives new writes (000000000123456789.log). When segment.bytes or segment.ms is hit, the segment rolls. Sparse offset and time indexes enable O(1) offset-to-position lookup. VaultCommerce sets log.segment.bytes=1GB and log.segment.ms=604800000 (7 days max per segment).

Key points

  • Segment — immutable file chunk of a partition log; active segment alone accepts writes
  • Log end offset (LEO) — next offset to be written on the leader
  • High watermark (HW) — offset replicated to all ISR; consumers see up to HW-1
  • retention.ms — time-based segment deletion policy
  • log.dirs — comma-separated data directories; VaultCommerce uses 3 NVMe mounts per broker

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Retention and disk planning
  • VaultCommerce rollout checklist

Test yourself

Answer these before moving on — recall is what makes it stick.