Why this matters
- VaultCommerce L1 on-call runs this checklist in under 10 minutes.
- Prevents premature offset resets and unnecessary broker restarts.
- Documents escalation path to platform team with captured evidence.
- Living runbook updated after every postmortem.
Kafka brokersJMX
Prometheus
Grafana + PagerDuty
First five minutes
Acknowledge page. Open Grafana Kafka dashboard. Note alert name and time. Check #deploys channel for correlated releases. Run cluster-id and quorum describe. If ActiveControllerCount != 1, escalate immediately to platform.
Key points
- Triage order — cluster → brokers → consumers → clients
- Evidence bundle — lag describe, log snippet, deploy correlation
- Escalation threshold — URP > 0 for 10m, controller loss, disk > 90%
- Safe actions — scale consumers (≤ partitions), throttle non-critical producers
- Unsafe without approval — offset reset, broker restart, retention change
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Before escalating or acting
- VaultCommerce rollout checklist
Test yourself
Answer these before moving on — recall is what makes it stick.