How PR analysis works
When an eval workflow finishes, evalship stores its results as a run for that commit, compares the run with a baseline, and reports the difference in two places: a comment on the pull request and a check named evalship on the commit.
Every number in the report is computed by deterministic code. A model is used for one thing only: naming failure patterns and their likely cause (see the last section of this page).
Baselines
| The run is for | Baseline |
|---|---|
| A pull request | The run for the pull request's merge-base commit. If there is none, the latest run on the pull request's base branch; if there is none, the latest run on the default branch. |
| A push to the default branch | The previous run on the default branch. |
| A push to another branch | The latest run on the default branch. |
Without a baseline, the report lists the results with no comparison and says it's the first run evalship has seen. That's why the eval workflow must run on pushes to the default branch.
Change categories
Cases are matched by suite and key.
| Category | Meaning |
|---|---|
| Newly failing | Passed in the baseline, fails now. |
| Regressed | Same pass or fail state, and the score dropped by more than the tolerance. |
| Newly passing | Failed in the baseline, passes now. |
| Improved | Same pass or fail state, and the score rose by more than the tolerance. |
| New | Not in the baseline. |
| Removed | In the baseline, not in this run. |
| Flaky | Would be one of the first four, but the case is flaky. |
| Unchanged | Everything else. |
The comment calls newly failing and regressed cases together "regressions", and newly passing and improved cases "improvements".
Tolerance
A score change counts only when the absolute difference is larger than the tolerance: 0.05 by default, configurable for the repository and per suite in .evalship.yml. With the default, a score going from 0.82 to 0.78 is unchanged; 0.82 to 0.76 is regressed. A flip between passing and failing always counts, whatever the score.
Flakiness
A case is flaky when, over its last 10 runs on the default branch, it either:
- flipped between passing and failing at least twice, or
- has at least 3 scores whose standard deviation is more than twice its suite's tolerance.
A flaky case that changes gets a :yellow_circle: row marked "flaky, ignored" in the status table and is listed under "Flaky, ignored" in the details. It stays out of regressions, improvements and the gate. The pass rate delta is then computed on the non-flaky cases. Set gate.ignore_flaky: false to turn this off.
The comment
evalship posts one comment per pull request and edits it in place on every run. It comments only on open pull requests, and only with results for the pull request's latest commit. If you delete the comment, the next run posts a new one.
This is a comment for a pull request that shortened a system prompt, with the details cut short:
<a href="https://evalship.com/gh/acme/support/pull/42"><img src="https://evalship.com/c/TOKEN.svg" alt="2 real regressions in support-bot, likely from the removed language rule in prompts/support_system.md." width="720"></a>
| | Eval | `main@a1b2c3d` | this PR |
|:-:|---|:-:|:-:|
| :red_circle: | `refund-nl-1` | :white_check_mark: 0.93 | :x: 0.41 |
| :red_circle: | `escalation-nl-2` | :white_check_mark: 0.88 | :x: 0.35 |
| :white_circle: | `app/refunds.py#RefundPolicy.explain` | new LLM call | no eval |
:mag: **2 of 2 new failures reply in English to Dutch messages**, likely from the removed language rule in `prompts/support_system.md`.
:moneybag: Eval run cost +39% ($0.41 to $0.57).
<details><summary>Details: 48 evals, 44 passed (-2 vs <code>main@a1b2c3d</code>)</summary>
...
</details>
<sub>[Full report](https://evalship.com/gh/acme/support/pull/42) ยท Was this useful? React with ๐ or ๐</sub>
The parts, in order: the card, the status table, the notes, any .evalship.yml problems, the details, and the footer.
Card
The image at the top links to the full report. Its alt text is the lead sentence, described under the check below.
| Part | What it shows |
|---|---|
| Status | In large type: "2 regressions", "No regressions, 1 untested change", "No regressions", or "First run: 46/48 passing" when there is no baseline. |
| Pass count | Passing evals out of the total, with the change against the baseline. |
| Notes | The baseline ref (vs main@a1b2c3d), how many flaky evals were ignored, and the untested changes when there are also regressions. |
| Suite bars | One bar per suite, up to 6, with passed and total and the change against the baseline. Each cell is one eval. A suite with no results in this run shows "no results here". |
| Cell | Meaning |
|---|---|
| Solid red | Regressed in this pull request. |
| Amber dither | Flaky, ignored. |
| Sparse red dither | Failing, but not broken by this pull request, for example already failing on the baseline. |
| Green dither | Passing. |
The card shows counts and suite names only: no case names, inputs or outputs. evalship serves it at /c/<token>.svg, a signed URL, so the cards of private repositories can't be guessed.
Status table
One row per change, worst first, up to 6 rows. The rest are listed in the details.
| Row | Meaning |
|---|---|
:red_circle: |
A regression: the eval newly fails, or its score dropped beyond the tolerance. |
:white_circle: |
An LLM call or prompt that the pull request adds ("new LLM call") or changes ("changed") and no eval exercises ("no eval"). See Untested changes. |
:yellow_circle: |
A flaky eval that changed, marked "flaky, ignored". |
:green_circle: |
An improvement: the eval newly passes, or its score rose beyond the tolerance. |
For evals, the last two columns show the result on the baseline and in this pull request: :white_check_mark: or :x:, followed by the score when there is one.
Notes
Up to 3 lines below the table, for what a table can't show:
| Note | Shown when |
|---|---|
:mag: Failure pattern |
The model found 2 or more new failures that fail for the same visible reason. Ends with the likely cause when the model found one; a likely cause without a pattern gets a note of its own. At most 2 patterns. |
:warning: Didn't run |
A suite that had results in the baseline has none in this run. |
:moneybag: Cost |
Total eval cost changed by 20% or more, or total eval duration by 50% or more. |
Details
Below the notes, a collapsed block starts with a summary like "Details: 48 evals, 44 passed (-2 vs main@a1b2c3d)". It contains:
- A table of up to 15 suites with passed cases, mean score and the change against the baseline.
- Lists of regressed, flaky, new and removed cases, and of LLM calls and prompts without evals, up to 10 each.
- All failing cases, when there is no baseline.
The full report on evalship lists every case.
Footer
The comment ends with a link to the full report and asks for a thumbs up or thumbs down reaction. evalship reads the reactions on each update and when the pull request closes, and records them as feedback on what the comment reported.
On public repositories, the first comment that isn't a one-liner also suggests the README badge, once per repository.
When nothing changed
When there is a baseline and there are no regressions, untested changes, suites that didn't run, cost shifts or improvements, the comment is one line with no card:
:white_check_mark: **evalship**: 48 evals, no changes vs `main@a1b2c3d`. [Details](...)
When some cases fail in both runs, the line says so: "48 evals, 2 failing, no changes vs main@a1b2c3d." New cases, removed cases and flaky changes alone don't count as a change. To skip the comment in this case, set comment.only_on_change: true; a comment that already exists is still updated.
The check and the gate
Every processed commit gets a check named evalship, whether or not it belongs to a pull request. Its title is the lead sentence in plain text, or the one-line summary when nothing changed. Its output has the lead sentence, the status table, the notes, the gate result, the details expanded, and any .evalship.yml problems.
Lead sentence
The lead sentence states the most important point:
| Situation | Lead sentence |
|---|---|
| No baseline | "48 evals, 46 passing. This is the first run evalship has seen for this repo; results on main become the baseline for pull requests." |
| Regressions | "2 real regressions in support-bot", with ", likely from ..." when the model found a likely cause |
| Untested changes, no regressions | "No regressions, but this PR changes 1 LLM call or prompt that no eval exercises." |
| Improvements only | "No regressions, 1 improvement." |
| Otherwise | "No regressions." |
When flaky cases changed, the lead ends with "1 flaky eval ignored."
Gate
The check is advisory by default: it concludes neutral, which never blocks a merge. It becomes a gate when .evalship.yml sets at least one of these:
| Key | Fails the check when |
|---|---|
gate.fail_on_new_failures: true |
Any non-flaky eval went from passing to failing. |
gate.max_pass_rate_drop: <points> |
The pass rate dropped by more than that many percentage points, measured on non-flaky evals when flaky ones are involved. |
comparison.suites.<name>.required: true |
That suite has no results for the commit. |
With a gate, the check concludes success or failure, and a failure lists its reasons: "Check failed: 2 evals went from passing to failing." To block merges, require the evalship check in a branch protection rule or ruleset. The check is also neutral when evalship hits an error and for private repositories that aren't enabled. Use "Re-run" on the check in GitHub to process the results again.
What the model does
When a run has evals that newly fail (passed in the baseline, fail now), evalship makes one call to Claude Haiku with:
- up to 20 of those cases: key, suite, and the first 600 characters of input and output and 400 of the failure message;
- the diffs of up to 5 changed files with LLM calls or prompts, up to 1,500 characters each.
The model returns failure patterns (groups of failing cases with a short label for what their outputs have in common) and, when the diff clearly supports it, the likely cause and the file it's in. evalship then validates the answer:
- A pattern must name at least 2 of the case keys it was given; unknown keys are dropped. At most 2 patterns are shown.
- The likely cause is dropped unless its file is one of the pull request's changed LLM or prompt files.
- Labels and causes that contain numbers are dropped. Counts and scores always come from evalship.
Runs without new failures make no call. Calls are capped at 300 per account per month. Over the cap, or when the call fails, the comment is posted without patterns and likely cause; everything else is unchanged. What is sent is described in Data and privacy.