Why this matters
- PolicyDesk produces more answers than a human can read every day. A judge is how you sample quality continuously.
- A judge that always returns 5 because the rubric says "be generous" will hide a regression.
- Interviewers ask both how the judge works and how you stopped it grading an unrepresentative demo set.
- Judges have biases. They prefer longer answers, assertive tone, and their own phrasing. You design around that.
What the judge is
You send a strong model three things: the user question, the answer to grade, and the reference context or gold answer. A rubric tells it what "good" means: correctness against the context, faithfulness, relevance, and whether the refusal was appropriate. It returns a score, often 1 to 5, and a short reason. You can also ask for a pairwise choice: which of two answers is better.
Use it beside human review, not instead of a sample of human review. The judge is consistent and cheap relative to a panel. It is not ground truth.
What makes the setup usable
- The rubric is specific. "Is every number in the answer present in the context?" beats "is this helpful?"
- The output is structured JSON: score, pass or fail, reason, and the claims it checked. Free prose is hard to aggregate.
- You can score one answer against context, or compare two answers. Pairwise is better when the absolute scale drifts.
- Cost stays far below a human panel, so you can run it on every change to the prompt and on a daily sample of production.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 3 sections
- The flow
- How to pick the examples
- What you can trust
Test yourself
Answer these before moving on — recall is what makes it stick.