Why this matters
- Heisenbugs in distributed systems reproduce only under partition, load, or specific shard routing — printf debugging across 40 services fails.
- Structured logs with trace-id are mandatory; ShardPay rejects PRs that add unstructured log lines to the transfer path.
- On-call runbooks start with "find the trace" — everything else follows from that anchor.
- Senior interviews expect a systematic workflow, not "I'd check the logs."
The debugging workflow
ShardPay's on-call runbook for transfer failures follows five steps: anchor on business ID, pull trace, correlate logs, check metrics at incident time, reproduce or mitigate.
Step 1: Anchor on a single request
Support tickets include transferId. If missing, search by merchant ID + timestamp in the API access log. Everything downstream keys off this one request — resist the urge to grep globally first.
Step 2: Find the trace
Jaeger search by baggage transferId=TXN-c8e2. If no trace exists, check sampling (was it a fast success trace discarded?) or broken propagation (orphan subtree in a different trace ID).
Step 3: Correlate logs
Click trace ID in Grafana → log panel filtered by trace_id. ShardPay's JSON logs include trace_id, span_id, service, shard_id, and level. Follow the error log line backward to the first WARN.
Step 4: Check metrics at timestamp
Overlay trace timestamp on golden signals dashboard. Was shard 7 saturated? Did deploy happen 5 minutes before? Metrics explain why the span was slow; the trace shows where.
Step 5: Mitigate, then reproduce
Mitigate first (scale pool, rollback deploy, route around bad shard). Reproduce in staging with traffic shadowing if root cause is unclear.
Key points
- Correlation ID — business identifier (transferId) plus technical identifier (trace_id) linking all observability signals. ShardPay sets both at the gateway; internal services must not generate new trace IDs mid-request.
- Structured logging — JSON with fixed fields, not string concatenation.
{"level":"ERROR","trace_id":"abc","shard_id":7,"msg":"debit timeout"}is searchable;"Error on shard 7"is not. - Three pillars correlation — traces show path and latency; logs show decision context; metrics show system-wide state at that moment. ShardPay's Grafana dashboards join all three by trace_id.
- Time alignment — ensure clock sync (NTP) across services. A 30-second skew between trace timestamps and metric timestamps sends on-call chasing the wrong minute.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Walkthrough: missing transfer report
- Advanced techniques
Test yourself
Answer these before moving on — recall is what makes it stick.