Metrics, JMX, Prometheus, and Lag Alerts

Critical broker metrics

UnderReplicatedPartitions, OfflineReplicaCount, ActiveControllerCount (must be 1). Request handler idle percent below 20% signals overload. VaultCommerce alerts on disk above 75% and LogFlushRateAndTimeMs p99 spike.

Consumer lag alerts

kafka_consumergroup_lag_sum grouped by group, topic. Derivative alert on lag growth rate. Burrow optional for multi-cluster. VaultCommerce uses Prometheus recording rules for lag_velocity_5m.

VaultCommerce rollout checklist

Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. Alert on under-replicated partitions immediately. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.

# VaultCommerce Prometheus alert rules excerpt
groups:
  - name: kafka
    rules:
      - alert: KafkaUnderReplicatedPartitions
        expr: kafka_server_replicamanager_underreplicatedpartitions > 0
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "Kafka under-replicated partitions on {{ $labels.instance }}"

      - alert: InventoryConsumerLagHigh
        expr: kafka_consumergroup_lag_sum{group="inventory-service"} > 10000
        for: 5m
        labels: { severity: warning }