Judging the judge

The problem

On April 24-25, 2025, OpenAI shipped a GPT-4o update meant to improve the model's "default personality." Within days, users reported the model had become excessively flattering: agreeing with bad ideas, validating risky decisions, praising mediocre work. OpenAI pushed a mitigation that Sunday night and completed a full rollback by Monday, about 24 hours of emergency operations across the whole deployment fleet. OpenAI's own account of the cause is specific: the update "focused too much on short-term feedback" (thumbs-up and thumbs-down signals from users) "and did not fully account for how users' interactions with ChatGPT evolve over time." (OpenAI, Sycophancy in GPT-4o; Expanding on what we missed)

Strip away the specifics and this is a judging problem. Thumbs-up is a human judgment signal, aggregated at scale, used to steer a model. It turned out to be a bad judge: it rewarded whatever felt good in the moment, not whatever was actually correct or useful. The same structural risk applies the moment you build an LLM to grade another LLM's output for your own product: your judge can be consistent, confident, and systematically wrong, and nothing about watching its scores go up will tell you that.

Why it is hard

An LLM-as-judge pipeline looks suspiciously circular the first time you build one: you are using a model to grade a model. A good backend engineer's instinct is to distrust anything that isn't a deterministic oracle, and that instinct is correct, but it is usually aimed at the wrong target. The real danger is not that judges are subjective. Humans are subjective graders too, and humans are what you are trying to approximate at scale. The real danger is that LLM judges have specific, measurable, non-obvious biases that a reasonable engineer would not suspect from casual spot-checking.

Zheng et al.'s foundational study showed that GPT-4, judging two chatbot answers side by side, flipped its preferred answer when the two answers were simply swapped in position, with no other change. Consistency under this swap was only 65.0% even for the best-performing judge they tested. The same study documented verbosity bias: judges systematically favor longer, more padded answers independent of actual quality. (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena) If you build a pairwise judge and never test it for position sensitivity, you are producing a number that looks like a measurement and behaves like a coin flip with a thumb on the scale.

It gets harder. A 2026 large-scale study ("Reliability without Validity") ran 541,000 judgments across 21 judges and 3 benchmarks and found that raw agreement between a judge and human labels overstates the judge's actual, chance-corrected discrimination ability by 33 to 41 percentage points. Worse, the same judge's relative ranking shifted by as much as 14 positions when moved from one benchmark to another. The paper names this the "consistency-bias paradox": a judge can be highly consistent with itself (reliable) while still being systematically wrong (not valid). (Reliability without Validity) A judge that always agrees with itself is not the same thing as a judge that is correct.

The idea: critique shadowing, bias audits, and reward signals

What makes a judge trustworthy

The fix Hamel Husain documents is called critique shadowing: before writing a single judge prompt, have a domain expert label a batch of real outputs pass or fail, and require a written critique explaining why for every label. You then build the judge's prompt from those critiques, often as few-shot examples, and iterate the prompt against the expert's labels until agreement is high. This replaces the common failure of 1-5 Likert-scale judges, whose scores are not actionable (what does a 3.4 average mean, and what do you fix to move it) and are not reproducible across small prompt changes. (Using LLM-as-a-Judge, Hamel Husain)

Bias types and their mitigations

Reward signals are judges too

The GPT-4o incident is the clearest illustration that a "judge" does not have to be an explicit LLM-as-judge pipeline. Any signal you optimize against is a judge, including an implicit one like a thumbs-up rate. OpenAI's fix was not a smarter model: it was recognizing that short-term approval and long-term usefulness are different things, and that optimizing hard against the former without an orthogonal guardrail eval for the exact failure mode you're afraid of (sycophancy, in this case) will eventually produce that failure mode at scale. The enterprise lesson generalizes beyond chat personality: if your support bot's only quality signal is "did the dealer say thanks," you have built the same trap.

Re-validation is not optional

Position and verbosity sensitivity, and judge-human agreement, are not properties you check once at launch. Alignment must be re-checked whenever the prompt, the model under test, or the product changes materially, the same way you would not trust a staging-to-production parity check that was last run six months ago.

In practice at Northwind

Northwind wants an automated judge for Desk's answers to dealer questions about return policy, because manually reading every transcript does not scale past a few hundred conversations a week.

1. Start with critique shadowing, not a prompt you write from memory. Have Northwind's support lead label 50-100 real Desk transcripts pass or fail, with a one-line critique for each fail ("cited the wrong return window," "correct policy, condescending tone"). This doubles as the error analysis work from Lesson 11; the two steps feed each other.
2. Build the judge prompt from those critiques as few-shot examples. Measure its agreement with the support lead's labels on a held-out slice before trusting a single score it produces in CI.
3. If the judge ever does pairwise comparisons (old prompt vs. new prompt on the same question), run every comparison in both orderings and treat disagreement between orderings as a red flag about the judge, not about the prompts.
4. Explicitly separate "did the dealer seem satisfied" (an implicit approval signal, prone to the same sycophancy trap OpenAI hit) from "was the answer factually and policy-correct" (a dedicated correctness check). Do not let a single blended score hide which one moved.
5. Put a recurring check on the calendar, tied to the same cadence as error analysis, that re-measures judge-human agreement, not just judge output trends. A judge that quietly drifts out of alignment with human judgment over six months will look stable in its own dashboard the entire time.

What strong engineers know

• A judge that agrees with itself 100% of the time across repeated runs has told you nothing about whether it agrees with the truth; reliability and validity are different properties and both must be checked. (Reliability without Validity)
• Pairwise LLM judges that have not been tested for position sensitivity on your own prompts are producing a number, not a measurement. (MT-Bench)
• Any metric you optimize against, including an implicit one like a thumbs-up rate, is a judge, and inherits all the same risks as an explicit LLM-as-judge pipeline. (OpenAI, Expanding on sycophancy)
• A 1-5 Likert score is rarely actionable; a binary pass/fail with a written critique tells you what to fix. Prefer the latter when building a judge from scratch.
• Judge quality rankings do not transfer across benchmarks; a judge validated on someone else's eval set is not validated on yours. (Reliability without Validity)
• Position-swap averaging and cross-model judging both double the judge's API cost. Budget for that cost explicitly rather than skipping the mitigation to save money.
• Judge-human alignment degrades silently over time as the prompt, model, and product evolve; schedule re-validation the same way you schedule a security review, not as a one-time launch gate.
• OpenAI's own GPT-4o rollback took about 24 hours once confirmed, not because the fix was hard, but because managing stability across a deployed fleet at scale is itself an operational cost; the cheaper intervention is catching the drift before 100% rollout, with a guardrail eval for the specific failure you fear.

Decide

Northwind's new Desk prompt, shipped last Friday, scores higher than the old prompt on the existing helpfulness judge (a single 1-5 LLM score). Dealer satisfaction replies in Slack also look more positive this week. You have one day before the change is considered final. Choose one path and justify it against the others:

(a) Ship it; both the automated judge and the qualitative Slack signal agree it's an improvement. (b) Hold the ship, and first check whether the new prompt is more agreeable on factually wrong or policy-violating dealer claims specifically, using a small adversarial set, before trusting the general helpfulness score. (c) Ask the judge to re-grade both prompts in the opposite order (new-then-old instead of old-then-new) to rule out position bias, and ship only if the preference holds both ways. (d) Replace the 1-5 helpfulness judge with critique shadowing against the support lead's own labels before making any ship decision, even if it delays the rollout past today.

One sentence summary

An LLM judge that agrees with itself is not the same thing as an LLM judge that is right, and the gap between those two is exactly where a sycophancy-style regression hides.

Check question

Northwind is deciding whether to use the same LLM judge to grade both Desk's Slack answers (internal employees) and Desk's dealer-facing email replies (external, written by non-technical users). The judge was validated only on the Slack transcripts. What would you check before trusting its scores on the email channel, and why does the MT-Bench ranking-shift finding matter here?

Sources

• Sycophancy in GPT-4o: what happened and what we're doing about it · OpenAI · Apr 29, 2025 · Primary postmortem on the rollback.
• Expanding on what we missed with sycophancy · OpenAI · 2025 · Rollout/rollback timeline and the "short-term feedback" root cause.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · Zheng et al., LMSYS/UC Berkeley · Jun 2023 · Position/verbosity bias, 65.0% consistency figure.
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models · Justin Norman et al. · Jun 2026 · 33-41pp agreement overstatement, 14-position rank shift.
• Using LLM-as-a-Judge For Evaluation: A Complete Guide · Hamel Husain · Oct 2024 · Critique shadowing method.
• Creating a LLM-as-a-Judge That Drives Business Results (commentary) · Simon Willison · Oct 30, 2024 · Independent confirmation of critique shadowing.