DNS, Service Discovery, and Health-Aware Routing

Finding healthy instances

In a dynamic fleet, the question "where is the ledger service?" has a different answer every few minutes. Pods start, stop, crash, and migrate. Hardcoded IP lists break on the first deploy. DNS provides name resolution but caches answers for seconds to minutes. Service discovery bridges the gap: a live registry that tracks which instances are healthy right now.

ShardPay ledger pods register on startup with Consul and deregister on graceful shutdown. The API tier resolves ledger.shardpay.internal to current healthy endpoints via a discovery client that refreshes every 5 seconds — faster than DNS TTL, slower than per-request lookup.

DNS TTL vs real-time discovery

DNS is simple and universal but stale. A TTL of 30 seconds means clients may send traffic to dead pods for up to 30 seconds after termination. For stateless API pods, retries mask this. For stateful ledger leaders, 30 seconds of writes to a dead leader causes data loss risk.

ShardPay uses a layered approach: Kubernetes DNS for coarse resolution (which service exists), Consul for fine-grained health-aware routing (which instances are ready), and client-side load balancing for shard-aware routing (which shard owns this account).

Walkthrough: deploy with stale DNS

ShardPay deploys a new ledger version. Kubernetes terminates old pods and starts new ones over 60 seconds. With DNS TTL = 30s, API clients cache old pod IPs for up to 30 seconds. During this window, ~4% of transfer requests fail with connection refused.

Fix: readiness-gated registration. New pods register with Consul only after passing health checks. Old pods deregister before termination (preStop hook). Consul removes unhealthy endpoints within 2 seconds. API clients refresh every 5 seconds. Failure rate during deploy drops to under 0.1%.

// ShardPay — discovery-backed gRPC client with health-aware refresh
public class LedgerDiscoveryClient {
    private final ConsulClient consul;
    private volatile List<Endpoint> healthyEndpoints = List.of();
    private final ScheduledExecutorService refresher = Executors.newSingleThreadScheduledExecutor();

    public LedgerDiscoveryClient(ConsulClient consul) {
        this.consul = consul;
        refresher.scheduleAtFixedRate(this::refresh, 0, 5, TimeUnit.SECONDS);
    }

    private void refresh() {
        healthyEndpoints = consul.getHealthyInstances("ledger", "v1");
    }

    public LedgerShardClient route(AccountId accountId) {
        List<Endpoint> endpoints = healthyEndpoints;
        if (endpoints.isEmpty()) throw new UnavailableException("no healthy ledger instances");

        Endpoint target = consistentHash.pick(endpoints, accountId);
        return grpcClient.forEndpoint(target);
    }
}

Health checks that matter

A health check that only verifies "process is running" is insufficient. ShardPay's ledger health check validates: (1) Raft node is part of a quorum, (2) replica lag is under 2 seconds, (3) disk write latency is under 10ms. A pod that is alive but cannot serve correct data is removed from the pool before it causes errors.

Liveness vs readiness distinction: liveness means the process should be restarted (it is stuck). Readiness means the instance should receive traffic (it is caught up and healthy). ShardPay restarts on liveness failure; removes from load balancer on readiness failure without restart.

Service mesh and discovery at scale

At ShardPay's scale (40 API pods, 18 ledger pods, 6 shards × 3 replicas), manual endpoint management is impossible. Kubernetes Endpoints controller watches pod labels and updates DNS automatically. Consul adds health-aware filtering on top. Istio service mesh (adopted in module 11) adds mTLS, retry policies, and circuit breaking at the discovery layer.

The principle remains: never hardcode endpoints. Every service name resolves through discovery. Every resolution is health-aware. Every deploy is graceful.