Why this matters
- StreamHub uses ML for content moderation (flag inappropriate streams), recommendation ranking (suggest videos), and thumbnail quality scoring — each needs a production inference path.
- A model that scores 95% accuracy in Jupyter is worthless if inference takes 5 seconds or crashes under 100 concurrent requests.
- GPU autoscaling, model versioning, and shadow deployments are production ML engineering, not data science.
Inference serving components
- Model registry — versioned model artifacts (weights, config, preprocessing) in S3 or MLflow.
- Inference server — TorchServe, Triton, or BentoML serving predictions via REST/gRPC.
- Feature store — pre-computed features for low-latency inference (Feast, Tecton).
- Autoscaling — scale GPU pods on queue depth or request rate; scale to zero for batch models.
- A/B routing — split traffic between model versions; compare accuracy and latency.
- Monitoring — prediction latency, throughput, data drift, model accuracy degradation.
Serving architecture
ML inference on AWS
# FastAPI inference endpoint
from fastapi import FastAPI
import torch
app = FastAPI()
model = torch.jit.load("models/content-moderator-v3.pt")
model.eval()
@app.post("/v1/moderate")
async def moderate(request: ModerateRequest):
features = preprocess(request.thumbnail_url, request.title)
with torch.no_grad():
logits = model(features)
scores = torch.softmax(logits, dim=-1)
return {
"model_version": "content-moderator-v3",
"scores": {
"safe": float(scores[0]),
"nsfw": float(scores[1]),
"violence": float(scores[2])
},
"action": "block" if scores[1] > 0.85 else "allow"
}
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 4 sections
- Kubernetes deployment with GPU
- Batching for throughput
- A/B model testing
- Scale estimates
Test yourself
Answer these before moving on — recall is what makes it stick.