Why this matters
- PolicyDesk traffic spikes on Monday morning and sleeps at night. A fixed replica count is either an outage or a wasted GPU bill.
- Interviewers want the HPA fields and the LLM-specific twist: CPU is a bad proxy if the real work is GPU or a remote model API.
- Scaling too fast flaps. Scaling on the wrong metric scales the wrong thing.
- The vector database does not scale just because the API pods did. Say that out loud.
What HPA does
The Horizontal Pod Autoscaler watches a metric and sets the replica count of a Deployment between minReplicas and maxReplicas. When the metric sits above the target, it adds pods. When it sits below, it removes them, after a stabilization delay so one quiet minute does not delete capacity you need again immediately.
For an LLM service this keeps latency reasonable as concurrent questions grow, and it lets you pay for fewer GPUs or fewer proxy pods when the office is empty.
HPA scales pods. It does not scale a single pod's GPU memory, and it does not speed up a hosted model vendor who is rate-limiting you. If the bottleneck is the vendor's quota, more pods only produce more errors.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 4 sections
- Fields you should be able to write
- The picture
- A manifest you can defend
- LLM-specific cautions
Test yourself
Answer these before moving on — recall is what makes it stick.