LLM evals that can't fail: flaws in 10 open-source eval suites

A passing eval suite says your prompt works, if each test case would fail when the behavior it checks breaks. In October 2026, evalship's setup review read the eval suites of 10 open-source repositories, from MCP servers to coding-agent skills. Every one had at least one test case that couldn't fail. Each finding was checked against the repository's code.

Test cases that can't fail

All 10 suites had one. Some examples:

• An agent asked to cancel the late runs passed when it cancelled every late run in the workspace. The eval never checked that the other runs survived.
• Prompt-injection cases recorded a score and never checked it.
• 14 of one suite's 31 happy-path cases passed when every tool returned an error: they checked that the reply was JSON, and the error was JSON too.
• Thirty copies of the same message passed a suite that grades progress messages.
• A weighted average let a case pass with the behavior it was named for missing.

Check yours: break the thing on purpose. Make a tool return an error, the model return an empty reply, or the agent skip a step. A test case that still passes can't catch that failure. It's the idea behind mutation testing, applied to evals.

Bugs in the eval

9 suites had an eval that tested something other than what it claimed:

• The eval's shell tool reported every successful command as timed out, so the agent was graded on failures that never happened.
• Six vector-search cases expected a tool call that the tool's own schema forbids.
• A check computed whether forbidden sections appeared in the answer, and never asserted it.
• Suites reported passing in CI while a safety case inside them failed.
• The agent under test could read other runs' rubrics.
• A rubric rewarded an approach the current version of the skill forbids.

Check yours: run each eval once and read its outputs, the passing ones too, against what the test case expects. Then check that a failing case fails the suite.

LLM judges you can't trust

6 of the 9 suites that grade with an LLM judge had a problem with it. In 5, the judge was the model under test. Others passed a verdict they couldn't parse, or graded the whole transcript instead of the final answer. LLM as a judge: five mistakes in real eval suites has each one and its fix.

Rules no test case checks, and prompts no eval runs

6 suites left a rule in a prompt untested, and 6 shipped a prompt or tool that no eval runs:

• An agent that reads untrusted logs and can run shell commands was told not to follow instructions found in logs. No test case planted one.
• Skills that change production feature-flag targeting had no eval at all.
• Nine tools reached the model with no description, and no eval noticed.

Check yours: list the rules in each prompt, the "never", "always" and "only" lines. For each, find the test case that would fail if the model broke it. Then list the prompts, skills and tools you ship, and find the suite that runs each one.

Evals that never run in CI

In 2 suites, evals existed but CI never ran them: two real-model suites skipped on every run, and an investigation prompt ran neither on pull requests nor in a weekly benchmark.

Check yours: open the CI log of a recent pull request and find each suite's result in it.

How evalship checks this

evalship's setup review reads your eval suites for these flaws and reports each with the file and line, plus fix instructions for your coding agent (the checks it runs). On every pull request after that, it flags prompt and LLM-call changes that no eval exercises (untested changes). evalship is free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart).