Review your evals

At setup, your coding agent reads every scorer, judge and prompt and finds where your evals fall short, each flaw with the line it's on and a fix to paste back into your agent.

See the flaws in Eliza's evals →How the review works →

evalship.com/gh/acme_demo/support-bot

We found 3 flaws across 44 test cases

1 CriticalEval bug

Replies that leak customer data still pass

Privacy is a fifth of the score, so a warm, helpful reply that reads out another customer's address scores 0.80 and passes.

Found in tests/evals/test_support_bot.py:44

score = 0.5 * helpfulness + 0.3 * tone + 0.2 * privacy

Fix instructions for your coding agent

Fix this flaw in our LLM evals, found by evalship's review: Replies that leak customer data still pass. Privacy is a fifth of the score, so a warm, helpful reply that reads out another customer's address scores 0.80 and passes. See tests/evals/test_support_bot.py:44. Score privacy on its own and fail the case on any leak, whatever the other scores.
Fix it in a pull request; evalship confirms the fix on it.

2 ImportantUntested instruction

Nothing checks the refund limit in the system prompt

If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give.

Found in prompts/support_system.md:12

Never promise refunds above $100

⚠ No eval tests a refund request above $100.

Fix instructions for your coding agent

Fix this flaw in our LLM evals, found by evalship's review: Nothing checks the refund limit in the system prompt. If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give. See prompts/support_system.md:12. Add cases for $150 and $1,000 refunds that expect a handoff.
Fix it in a pull request; evalship confirms the fix on it.

3 WarningEval bug

Evals pass when the judge is down

When the judge call fails, the case scores 1.0, so a run without a working judge is all green.

Found in tests/evals/conftest.py:24

return 1.0  # don't fail the build on judge errors

Fix instructions for your coding agent

Fix this flaw in our LLM evals, found by evalship's review: Evals pass when the judge is down. When the judge call fails, the case scores 1.0, so a run without a working judge is all green. See tests/evals/conftest.py:24. Fail the case on a judge error, or skip it and count the skips.
Fix it in a pull request; evalship confirms the fix on it.

What the review looks for

Check	What it catches
Eval bug	A flaw in how an eval decides pass or fail: the answer leaks into the prompt, an average hides a failing hard rule, errors or skips count as passes, the scorer sees truncated or wrong input, or correct answers fail.
Untested instruction	A rule in a prompt, such as "never promise refunds above $100", that no test case checks.
Judge problem	An LLM judge you can't trust: it grades its own model's output, is swayed by the answer's claims, or uses an undefined scale. Or a suite only judges grade, where code checks would catch the same failures.
Untested prompt	Every prompt and AI call no suite exercises, in one flaw evalship makes from the review. Skip the ones not worth an eval.
Evals never run in CI	Evals exist, but CI never runs them, or not on pull requests.
Suspiciously easy suite	Test cases that can't realistically fail, such as checking only for non-empty output, or a threshold of 0.
Thin or skewed dataset	Few test cases, near-duplicates, no edge or adversarial cases, or one language or document layout while the product handles several.
Nondeterminism	Temperature above 0 without repeats or tolerances, live network calls, or no seeds.

After setup, a pull request that fixes a flaw says so in its comment, and Review marks the flaw fixed once it merges.

See what evalship finds in your evals

Ask your coding agent:

Set up https://evalship.com/setup.md