Why this matters
- VaultCommerce upgraded 3.6 → 3.7 KRaft with zero client downtime.
- Upgrade order: controllers one at a time, then brokers; verify quorum between each.
- Inter-broker protocol and log message format version bump after all nodes updated.
- Kafka 4.x removes ZK — migration checklist is mandatory reading.
Rolling restart procedure
- Verify cluster healthy (no URP). 2) Upgrade binary on broker N. 3) Restart broker N. 4) Wait ISR sync. 5) Repeat. VaultCommerce maintains runbook with 15-minute soak between nodes. Never restart >1 controller simultaneously.
Key points
- Rolling restart — one broker at a time to preserve quorum
- inter.broker.protocol.version — wire protocol between brokers
- log.message.format.version — on-disk message format
- KRaft migration — ZK to KRaft one-way path with dual-write phase
- Soak period — validation window before next node restart
Post-upgrade version bump
kafka-features.sh / inter.broker.protocol.version and log.message.format.version after all nodes on new version. VaultCommerce runs kafka-broker-api-versions.sh to confirm uniform versions before bump.
VaultCommerce rollout checklist
Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. One broker at a time; verify ISR between restarts. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.
# VaultCommerce upgrade runbook — verify before/after each broker restart
kafka-metadata-quorum.sh --bootstrap-server kafka-1:9092 describe --status
kafka-broker-api-versions.sh --bootstrap-server kafka-1:9092 | grep -i version
# After all brokers on 3.7.0, bump protocol (example)
kafka-configs.sh --bootstrap-server kafka-1:9092 \
--entity-type brokers --entity-name 1 \
--alter --add-config inter.broker.protocol.version=3.7,log.message.format.version=3.7
Quick recall
Everything you need if you only revisit this box.
- One broker at a time; verify ISR between restarts.
- Never restart majority of controllers together.
- Bump protocol version only when all nodes upgraded.
Test yourself
Answer these before moving on — recall is what makes it stick.