Why this matters
- PolicyDesk, the HR web app, and a batch summarizer should not each embed a different copy of the model client.
- Interviewers ask this to see if you can separate application logic from inference: validation, batching, streaming, and metrics.
- A stateless service is what you scale. The model weights are the thing you cache, not the user's session.
- Swapping a model should not force every client to change its URL or its JSON.
What the service is
An inference microservice loads a model, or holds a client to a model runtime, and exposes an API that generates text for a prompt. It owns hardware choices, batching, timeouts, and logging. A chatbot, a RAG API, or a backend job calls it the same way.
"Loads a model" can mean the process holds GPU weights (vLLM, TGI, Ollama in a sidecar) or the process calls a hosted API (OpenAI, an internal gateway). The microservice boundary still helps in the hosted case: one place for retries, redaction, budgets, and a stable schema when you switch vendors.
What callers get
- A simple contract: prompt in, text out, optionally streamed.
- Freedom from tokenizer details, GPU drivers, and model-file paths.
- One security and quota layer instead of keys scattered through every app.
- The ability to run several replicas behind a load balancer.
Characteristics worth listing
What a serious service includes
- An HTTP API such as
POST /v1/chatorPOST /v1/generate, with a documented JSON schema. - More than one backend. The same route can target a hosted model, a Hugging Face weight running locally, or Ollama, chosen by configuration.
- Request checks. Maximum prompt length, allowed parameters, authentication. Reject what you will not pay to run.
- Stateless replicas. Any instance can serve any request. Conversation state, if you need it, lives in a store, not in the process memory of one pod. The KV cache is an exception you design for: it is an optimization tied to a request or a session affinity choice, not your source of truth.
- Streaming, batching, logs, and metrics. Streaming for interactive UX. Batching for throughput. Logs and metrics so you can see latency and token counts.
- A place to run. GPU nodes when you host weights. Ordinary CPU nodes when you only proxy a hosted API. A model cache on disk so replicas do not re-download weights on every start.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 3 sections
- The boxes
- How you would build it
- Why bother with the extra process
Test yourself
Answer these before moving on — recall is what makes it stick.