Why this matters
- VaultCommerce order events use 7-day delete retention; customer profiles use compaction.
- Tombstones (null values) delete keys in compacted topics after
delete.retention.ms. - Wrong policy fills disks or loses replay history.
- Compaction underpins Kafka Streams state store changelog topics.
Time and size retention
retention.ms=604800000 (7 days) on order topics — enough for replay and incident recovery. retention.bytes caps per-partition size as safety valve. VaultCommerce never relies on retention alone for compliance deletion — PII scrubbing happens in consumers.
Key points
- cleanup.policy=delete — time/size based segment removal
- cleanup.policy=compact — key-level retention of latest value
- Tombstone — record with null value signaling key deletion
- delete.retention.ms — how long tombstones survive before purge
- Compacted changelog — backing topic for Kafka Streams state stores
Log compaction
cleanup.policy=compact retains latest value per key. Customer email updates overwrite prior records. Tombstone with null value removes key after compaction cycle. min.compaction.lag.ms delays compaction so consumers see all updates in order first.
VaultCommerce rollout checklist
Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. Delete for event streams; compact for changelogs. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.
# VaultCommerce — convert profile topic to compacted (new topic + mirror, not in-place)
kafka-topics.sh --bootstrap-server kafka-1:9092 --create \
--topic vaultcommerce.customers.profile.v1 \
--partitions 6 --replication-factor 3 \
--config cleanup.policy=compact \
--config min.compaction.lag.ms=3600000 \
--config delete.retention.ms=86400000
kafka-configs.sh --bootstrap-server kafka-1:9092 \
--entity-type topics --entity-name vaultcommerce.customers.profile.v1 --describe
Quick recall
Everything you need if you only revisit this box.
- Delete for event streams; compact for changelogs.
- Tombstones remove keys from compacted topics.
- Never compact multi-event lifecycles on one key.
Test yourself
Answer these before moving on — recall is what makes it stick.