Skip to main content
PreparationZone Logo
PrepZone

Golden Signals and Tail Latency

Latency, traffic, errors, saturation — why p99 matters more than average for ShardPay.

Pass checkpoint below

Why this matters

  • Average latency hides the 1% of transfers merchants remember — p99 and p999 matter more than mean.
  • Saturation predicts incidents before error rates spike: a connection pool at 95% fills before timeouts appear in SLIs.
  • Dashboards that fit one screen keep on-call effective; ShardPay's transfer war room has exactly four panels.
  • Interviewers ask you to pick metrics for a new service — golden signals are the structured starting point.
Latencyp50 / p99
TrafficRequests/sec
Errors5xx rate
SaturationCPU, pool, queue
Latency, traffic, errors, and saturation — the four signals Google SRE recommends for every service.

The four golden signals

Each signal answers one question about system health. Together they cover user experience and capacity headroom.

Latency — how long requests take. ShardPay tracks p50, p95, p99, and p999 for transfers at the gateway. A rising p50 suggests systemic slowdown; a rising p999 with stable p50 suggests tail outliers (GC pauses, hot shards).

Traffic — demand on the system. ShardPay measures transfers/sec, API requests/sec, and Kafka consumer lag. Traffic spikes during Black Friday are expected; unexpected 3x traffic may indicate a retry storm or attack.

Errors — rate of failed requests. ShardPay splits HTTP 5xx (server fault), 4xx business rejections (insufficient funds), and timeout errors (client or server). Alerting on 5xx rate, not total error rate — business rejections are healthy behavior.

Saturation — how full resources are. CPU, memory, disk I/O, connection pools, thread pools, Kafka consumer group lag. ShardPay pages when any shard's JDBC pool exceeds 85% for 5 minutes — before timeouts affect the SLI.

Key points

  • Latency distribution — never alert on mean alone. ShardPay's p99 transfer latency SLO is 200ms while p50 is 45ms; mean of 60ms would hide p99 violations.
  • Traffic as context — errors during low traffic may be benign (deploy glitch); errors during peak traffic are merchant-visible. ShardPay normalizes error rate against traffic volume in dashboards.
  • Error taxonomy — separate infra errors from business errors. A 402 Insufficient Funds is not an SLO miss; a 503 Shard Unavailable is.
  • Saturation as leading indicator — pool exhaustion, disk 90%, CPU throttling predict errors 5–15 minutes ahead. ShardPay's most prevented incidents came from saturation alerts, not error alerts.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • ShardPay transfer dashboard
  • Alerting philosophy

Test yourself

Answer these before moving on — recall is what makes it stick.