Debugging Distributed Failures

The debugging workflow

ShardPay's on-call runbook for transfer failures follows five steps: anchor on business ID, pull trace, correlate logs, check metrics at incident time, reproduce or mitigate.

Step 1: Anchor on a single request

Support tickets include transferId. If missing, search by merchant ID + timestamp in the API access log. Everything downstream keys off this one request — resist the urge to grep globally first.

Step 2: Find the trace

Jaeger search by baggage transferId=TXN-c8e2. If no trace exists, check sampling (was it a fast success trace discarded?) or broken propagation (orphan subtree in a different trace ID).

Step 3: Correlate logs

Click trace ID in Grafana → log panel filtered by trace_id. ShardPay's JSON logs include trace_id, span_id, service, shard_id, and level. Follow the error log line backward to the first WARN.

Step 4: Check metrics at timestamp

Overlay trace timestamp on golden signals dashboard. Was shard 7 saturated? Did deploy happen 5 minutes before? Metrics explain why the span was slow; the trace shows where.

Step 5: Mitigate, then reproduce

Mitigate first (scale pool, rollback deploy, route around bad shard). Reproduce in staging with traffic shadowing if root cause is unclear.

Walkthrough: missing transfer report

Merchant M-3301 reports a $2,400 transfer " disappeared" — debited from their account but not credited to the recipient 20 minutes ago.

1. Trace lookup: transferId=TXN-f1a9 → trace shows saga reached DEBITED on shard 2, credit call to shard 6 returned HTTP 504 after 30s.
2. Logs: orchestrator log at 18:14:02 — CreditException: gateway timeout shard-6; saga state persisted as COMPENSATING.
3. Metrics: at 18:14, shard 6 pool utilization was 99% — saturation caused timeouts, not logic bug.
4. Resolution: compensation completed at 18:14:35 — debit reversed. Merchant saw funds return but no explanation.
5. Root cause: shard 6 pool undersized for end-of-day batch overlap with transfers. Fix: separate pool for batch vs API, increase max connections.

// ShardPay structured logging — MDC populated from OTel context
@Slf4j
public class TransferOrchestrator {
    public void creditShard(SagaInstance saga) {
        MDC.put("transfer_id", saga.transferId());
        MDC.put("saga_id", saga.sagaId());
        MDC.put("shard_id", String.valueOf(saga.toShard()));
        try {
            log.info("credit attempt started amount_cents={}", saga.amountCents());
            shardClient.credit(saga.toShard(), saga.request());
            log.info("credit succeeded");
        } catch (TimeoutException e) {
            log.error("credit timeout after_ms={}", 30_000, e);
            compensateDebit(saga);
        } finally {
            MDC.clear();
        }
    }
}

Advanced techniques

When the standard workflow fails — intermittent bugs, race conditions, partition-specific failures — ShardPay escalates to traffic shadowing and chaos reproduction.

Traffic shadowing

Production traffic is mirrored to staging with synthetic account IDs. Staging runs the same code path without affecting real balances. ShardPay caught a race condition in saga recovery that only appeared at >500 transfers/sec — invisible in unit tests, visible in shadow traffic.

Partition and failure injection

After mitigation, reproduce in a controlled environment. ShardPay's staging cluster runs Toxiproxy between orchestrator and shards to simulate 504 timeouts. Confirm compensation fires correctly before declaring fix verified.

Postmortem discipline

Every Sev-2+ incident produces a blameless postmortem with timeline, root cause, action items, and runbook updates. ShardPay's missing-transfer report above updated the runbook: check saga state in orchestrator DB before assuming data loss.