Why this matters
- Hardcoded IPs break the moment autoscaling replaces a pod.
- DNS TTL vs real-time discovery is a classic trade-off.
- ShardPay uses discovery with health checks, not static lists.
- Stale endpoints cause silent traffic loss during deploys and failures.
DNS lookup path (Route 53)
Finding healthy instances
In a dynamic fleet, the question "where is the ledger service?" has a different answer every few minutes. Pods start, stop, crash, and migrate. Hardcoded IP lists break on the first deploy. DNS provides name resolution but caches answers for seconds to minutes. Service discovery bridges the gap: a live registry that tracks which instances are healthy right now.
ShardPay ledger pods register on startup with Consul and deregister on graceful shutdown. The API tier resolves ledger.shardpay.internal to current healthy endpoints via a discovery client that refreshes every 5 seconds — faster than DNS TTL, slower than per-request lookup.
Discovery mechanisms
- DNS — resolves a hostname to one or more IP addresses, cached by clients and resolvers with a TTL (time to live). ShardPay's Kubernetes internal DNS has a 30-second TTL; during a rolling deploy, up to 30 seconds of traffic may hit terminated pods.
- Service discovery — dynamic registry (Consul, etcd, Kubernetes Endpoints) updated in real time as instances register and deregister. ShardPay's Consul checks update within 2 seconds of a pod failing health checks.
- Health check — periodic probe (HTTP
/health, TCP connect, gRPC health protocol) that determines whether an instance is ready for traffic. ShardPay's ledger readiness probe checks Raft leader status and replica lag, not just process alive. - Client-side load balancing — the client fetches the instance list from discovery and picks one (round-robin, least-connections, consistent hash). ShardPay's gRPC client-side LB routes by
accountIdhash to the correct ledger shard.
DNS TTL vs real-time discovery
DNS is simple and universal but stale. A TTL of 30 seconds means clients may send traffic to dead pods for up to 30 seconds after termination. For stateless API pods, retries mask this. For stateful ledger leaders, 30 seconds of writes to a dead leader causes data loss risk.
ShardPay uses a layered approach: Kubernetes DNS for coarse resolution (which service exists), Consul for fine-grained health-aware routing (which instances are ready), and client-side load balancing for shard-aware routing (which shard owns this account).
Walkthrough: deploy with stale DNS
ShardPay deploys a new ledger version. Kubernetes terminates old pods and starts new ones over 60 seconds. With DNS TTL = 30s, API clients cache old pod IPs for up to 30 seconds. During this window, ~4% of transfer requests fail with connection refused.
Fix: readiness-gated registration. New pods register with Consul only after passing health checks. Old pods deregister before termination (preStop hook). Consul removes unhealthy endpoints within 2 seconds. API clients refresh every 5 seconds. Failure rate during deploy drops to under 0.1%.
// ShardPay — discovery-backed gRPC client with health-aware refresh
public class LedgerDiscoveryClient {
private final ConsulClient consul;
private volatile List<Endpoint> healthyEndpoints = List.of();
private final ScheduledExecutorService refresher = Executors.newSingleThreadScheduledExecutor();
public LedgerDiscoveryClient(ConsulClient consul) {
this.consul = consul;
refresher.scheduleAtFixedRate(this::refresh, 0, 5, TimeUnit.SECONDS);
}
private void refresh() {
healthyEndpoints = consul.getHealthyInstances("ledger", "v1");
}
public LedgerShardClient route(AccountId accountId) {
List<Endpoint> endpoints = healthyEndpoints;
if (endpoints.isEmpty()) throw new UnavailableException("no healthy ledger instances");
Endpoint target = consistentHash.pick(endpoints, accountId);
return grpcClient.forEndpoint(target);
}
}
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Health checks that matter
- Service mesh and discovery at scale
Test yourself
Answer these before moving on — recall is what makes it stick.