LLM as a judge: five mistakes in real eval suites

An LLM judge grades what code can't: whether an answer is correct, follows the instructions, or stays on topic. Few suites test the judge itself. In October 2026, evalship's setup review read the eval suites of 10 open-source repositories. 9 grade with an LLM judge. In at least 6 of the 9, the judge had one of the problems below. Each finding was checked against the repository's code.

Mistake	Suites, of 9
The judge is the model under test	5
A failed or unreadable verdict counts as a pass	2
The judge never sees the question	1
The judge reads the whole transcript, not the answer	1
A judge where a comparison would do	1

The judge is the model under test

In 5 suites, the model that answered also graded the answer. Two left the judge on a default: a flag that defaults to the same model as the one under test, and a CLI call with no model set. One CI workflow graded with the model under test unless a repository variable was set, and it wasn't. One tested a small 2024 model and graded with that same model. One judged messages on the model tier that writes them.

Models recognize their own text and rate it higher (Panickssery et al., 2024). The MT-Bench study calls this self-enhancement bias, next to position and verbosity bias (Zheng et al., 2023). A model also shares its own blind spots: if it misreads a question when answering, it misreads it the same way when grading.

Fix: name the judge in the config, a different model than the one under test, and from another family if you can. Fail the run when the two are the same.

A failed or unreadable verdict counts as a pass

In one suite, a judge reply without JSON made the helper return nothing, and the test skipped its assertion, so it passed. Criteria the judge marked "unknown" left the count. With a pass mark of 0.5, two of four criteria were enough, even when the two that failed were that the feature worked and the code had no syntax errors.

In another, the judge returned a list of checks and an overall pass. The code read only pass. A reply with a failed check still passed when it said pass: true.

Fix: decide the verdict in code, from the criteria. A reply that doesn't parse is never a pass: fail the test, or report it as couldn't run. One suite in the set does exactly this, and says so in a comment.

The judge never sees the question

One grader got the answer and the assertions, but not the question that was asked. "The answer is correct" then means "the answer sounds correct".

Fix: give the judge the input, the answer, the reference answer if you have one, and the rubric.

The judge reads the whole transcript, not the answer

One agent suite sent the judge every message of the run: earlier turns from the fixture's conversation history, the agent's intermediate messages, and raw tool output. The rubric told the judge that an expected element found anywhere in the output counts. A run whose final answer was wrong could pass, because the right fact appeared in a tool result.

Fix: grade the final answer. Check tool calls and their arguments with code.

A judge where a comparison would do

One prompt said: if there are no bugs, return the code unchanged. Its eval asked a judge whether the output was functionally equivalent to the input. A string comparison answers that exactly, for free, and never disagrees with itself between runs.

Fix: check with code what code can check: exact values, JSON shape, which tool was called. Use the judge for the rest.

And one from CI: the judge model retires

On PrefectHQ/prefect-mcp-server#200, 14 of 20 evals failed on the first push, all with the same 404: the judge model had been retired. Count those as couldn't run, not as failures. When you move to a new judge, scores before and after measure different things, so compare from the change on.

How evalship checks this

evalship's setup review reads your eval suites and reports each of these as a Judge problem, with the file and line, next to the other flaws it finds. On pull requests, it says when a change swaps the judge or the model, so results before and after aren't compared as if nothing changed (how the PR comment works). evalship is free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart).