Skip to main content
PreparationZone Logo
PrepZone

On-Call Runbook and Command Cheat Sheet

The commands and checks every VaultCommerce on-call engineer runs before escalating a Kafka incident.

Pass checkpoint below

Why this matters

  • VaultCommerce L1 on-call runs this checklist in under 10 minutes.
  • Prevents premature offset resets and unnecessary broker restarts.
  • Documents escalation path to platform team with captured evidence.
  • Living runbook updated after every postmortem.
Kafka brokersJMX
Prometheus
Grafana + PagerDuty
JMX metrics exported to Prometheus. Lag alerts route to on-call. Logs correlate via trace IDs.

First five minutes

Acknowledge page. Open Grafana Kafka dashboard. Note alert name and time. Check #deploys channel for correlated releases. Run cluster-id and quorum describe. If ActiveControllerCount != 1, escalate immediately to platform.

Key points

  • Triage order — cluster → brokers → consumers → clients
  • Evidence bundle — lag describe, log snippet, deploy correlation
  • Escalation threshold — URP > 0 for 10m, controller loss, disk > 90%
  • Safe actions — scale consumers (≤ partitions), throttle non-critical producers
  • Unsafe without approval — offset reset, broker restart, retention change

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Before escalating or acting
  • VaultCommerce rollout checklist

Test yourself

Answer these before moving on — recall is what makes it stick.