Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

What evalship checks

evalship checks in three places: your evals at setup, each change on every pull request, and every AI call your app makes in production.

  • At setup, the review asks whether your evals can catch a bad change at all.
  • On every pull request, evalship compares the eval results with your default branch's. It asks whether everything that changed is tested, whether the change broke anything, and whether the results can be trusted enough to say so.
  • In production, Monitor flags the AI calls that fail, overpromise, share private data or break a rule your evals test.

Code computes every number in the pull request checks. Rules settle failure patterns, likely causes and fixed review flaws wherever they can. A model writes the notes, and names a pattern, a cause or a fix only when the rules can't.

The review, once at setup

Your coding agent reads every scorer, judge and prompt, and reports every flaw that holds up, most valuable first. Low-stakes ones are folded away below the rest. Each one cites a file, a line and an exact quote, and evalship checks every citation against your repository: a flaw whose citation doesn't hold is hidden. See The review.

Check What it catches
Eval bug A flaw in how an eval decides pass or fail: the answer leaks into the prompt, an average hides a failing hard rule, errors or skips count as passes, the scorer sees truncated or wrong input, or correct answers fail.
Untested instruction A rule in a prompt, such as "only promise what the refund system allows", that no test case checks. Why it matters
Judge problem An LLM judge you can't trust: it grades its own model's output, is swayed by the answer's claims, or uses an undefined scale. Or a suite only judges grade, where code checks would catch the same failures.
Untested prompt Every prompt and AI call no suite exercises, in one flaw evalship makes from the review. Skip the ones not worth an eval.
Evals never run in CI Evals exist, but CI never runs them, or not on pull requests.
Suspiciously easy suite Test cases that can't realistically fail, such as checking only for non-empty output, or a threshold of 0.
Thin or skewed dataset Few test cases, near-duplicates, no edge or adversarial cases, or one language or document layout while the product handles several.
Nondeterminism Temperature above 0 without repeats or tolerances, live network calls, or no seeds.
Cost and speed The most expensive suites, repeated model calls, no caching, or a large judge model where a small one would do.
Mixed or duplicated setup Two frameworks testing the same thing, or dead eval files.

Each flaw comes with what to paste into your coding agent to fix it. The review keeps working after setup: a pull request that fixes a flaw says so in its comment, and Test marks the flaw fixed once it merges.

Why untested rules matter

What your bot tells a customer, your company said. Courts have ruled that way:

  • Canada, 2024. Air Canada's chatbot told a customer they could claim a bereavement fare after flying, which the airline's policy didn't allow. A tribunal rejected the argument that the chatbot was responsible for its own words, and ordered Air Canada to pay $812.02. Moffatt v. Air Canada, 2024 BCCRT 149
  • Germany, 2026. A clinic's chatbot gave its doctors specialist titles they didn't hold. The Higher Regional Court of Hamm treated the chatbot's answers as the company's own, even if the bot had only correct data to go on, and ordered the claims stopped. An appeal to the Federal Court of Justice is allowed. OLG Hamm, 4 UKl 3/25

A rule in a prompt holds only as often as the model follows it, and a prompt edit can drop it without a failing eval. So the review flags each rule no test case checks. When code could enforce the rule, such as a refund limit, its fix says to enforce it there, and the test case checks that the reply says what the code decided. On a pull request, a failure that would commit you this way is marked a liability risk.

AI is never wrong logs more failures like these, each with its sources.

On every pull request

Is everything that changed tested?

Check What it does
Prompt coverage Comes first in the comment, with a badge for the whole repository's AI calls and prompts. Lists the AI calls, prompt files and tool descriptions the pull request adds or changes that no eval exercises, with a prompt for your coding agent to add one. See Untested changes.
Not run on pull requests Says when only suites run by hand, on a schedule or by label exercise a change.
Not run here Says when a suite that exercises a change ran on the baseline but sent no results for this commit: a path filter missed the change, or its job crashed.
Not checked Says when the pull request changes a suite from your review, or the code it tests, and that suite sends evalship no results.
Eval flaws The model reads the diff and your review, and flags evals that can't catch the change, expected answers loosened in the same pull request as the prompt they grade, new suites CI never runs, and changes that repeat a review flaw.

With gate.fail_on_flaws, the check fails on a changed AI call or prompt that has no eval or whose suites sent no results, and on an open eval flaw, until each is fixed or dismissed. A flaw first found on a later push, in a file that push didn't change, shows on the report without failing the check: a fix is held up only by flaws in what it changed. Your coding agent can propose a dismissal with its reason: it shows on the report, and the check stays failed until someone with push access accepts it.

Did it break anything?

Check What it does
Regressions Finds the evals that passed on the base branch and fail now, and those whose score dropped by more than the tolerance (0.05 by default).
Beyond chance Counts a suite's failures as regressions only when it got worse beyond the noise measured over its last 30 runs on the default branch. The rest are listed as within chance, with a suggestion to rerun them. With the gate on, evalship reruns the workflow once for an eval that never fails on the default branch, and the check waits for that. In a suite with fewer than 3 runs there, an eval that got worse is too early to tell until it fails twice.
Flaky evals Sets aside evals that passed and failed on the same code, flipped between passing and failing at least twice in their last 10 runs on the default branch, or whose scores swing widely. They never count as regressions or fail the gate, and the baseline step gives your coding agent instructions to fix them. An eval that failed 3 or more runs in a row broke instead: it's listed to fix while it fails, and passing again after isn't a flip.
Repeats Judges an eval that runs several times on all its runs, with Fisher's exact test: 3 of 3 to 1 of 3 is within chance, 10 of 10 to 6 of 10 is a regression, with a stricter bar when many evals have repeats.
Failure patterns When 2 or more new failures fail for the same visible reason, names the pattern and the change that likely caused it, and the risk when the failing replies carry one: liability, privacy or safety.
New evals that fail Calls out new evals that fail, first: they have no result on the default branch to compare with.
Eval changed When a pull request changes only evals, CI workflows or docs, the evals it makes fail are marked eval changed, not regressions, and don't fail the gate: a stricter eval failing is the fix working.

Can the results be trusted?

Check What it does
Evals that couldn't run Leaves out of the comparison evals that never reached the model: a missing API key (pull requests from forks and Dependabot get no secrets), a rate limit, a timeout, a provider or judge error, or an output that is an HTTP error page, even if it passed. When every eval in a suite fails with the same error, it blames the harness, not the answers.
Evals that didn't run Notes a suite that had results on the base branch and has none here. Evals a run skipped on purpose (a tag, a filter, a path-filtered job) are marked not run, not failed.
Scoring changed Warns when the pull request rewrites the line where a suite decides pass or fail, its scorer, or its test cases' expected answers, so before and after may not compare. Only when a result moved, or when that suite doesn't run on pull requests.
Model swapped Warns when the pull request changes the model an eval runs or judges with. Only when a result moved.
Eval workflow changed Warns when the pull request deletes the workflow that runs the evals, removes its pull_request trigger, removes the results upload or its if: always(), or turns a step off with if: false, and when GitHub skipped an eval workflow the pull request adds or changes.

What did it cost?

Check What it does
Cost and duration Notes when the total eval cost moves by 20% or more, beyond what the same evals cost in the default branch's last 10 runs, or the duration by 50% or more.

See How the PR comment works for how each check shows up, and Configuration for the tolerance, the flaky rule and the gate.

In production

Monitor checks every AI call your app sends it, and emails your organization's owners when one opens an issue.

Check What it flags
Your eval rules An answer that fails a check your evals test for that call.
Replies An overpromise, a made-up fact, private data, an unsafe answer, a refusal or no answer.
Errors A call or a step under it that ended in an error, and an agent that keeps retrying a failing tool call.
Users A user who says an answer was wrong or useless, or asks for a person.

See What Monitor flags for each one, and Connect production to start.