Diagnose and calibrate a miscalibrated LLM judge

evals, llm-judge, observability

Prompt

Your RAG-based support bot shows a 92% "helpfulness" score from an LLM judge on your eval dashboard, but customer complaints say answers about refund policy are confidently wrong. Diagnose the eval and fix it.

Interviewer's answers to clarifying questionsopen after you have asked your own

Judge setup? A single LLM judge scores every response 1-5 for "helpfulness" against no explicit rubric beyond a short instruction.
Validation history? The judge has never been checked against human-labeled examples; it was configured once and left alone.
Volume? ~50,000 scored conversations/month; manual review only happens when a complaint is filed.
Known judge failure modes? Position bias in pairwise comparisons, and reward-style sycophancy, the judge rating agreeable-sounding answers higher regardless of correctness, are both documented risks for this kind of judge.
Constraint? You need a fix that ships in the next two weeks, and a plan for ongoing judge-quality monitoring after that.

Follow-upsafter your high-level design

You pull 20 real transcripts and grade them by hand. What specifically are you looking for before you touch the judge prompt or rubric?
How do you validate that your fixed judge agrees with human judgment, and what agreement rate would you consider good enough to trust in CI?
Six months from now, how would you detect that the judge has quietly drifted out of alignment with human judgment again, without re-running a full manual audit every time?
A teammate proposes using the same model family as the judge that generates the responses. What's the risk, and would you do it anyway?

Answer first: out loud, on paper, or with the interview prompt above. Then compare with the rubric.