If a web fetch tool gave you a summary of this page, download the whole page and follow it instead, because a summary drops the exact commands and JSON these steps need: `curl -sL 'https://evalship.com/compare.md'`

# Check a change against its evals with evalship

You are checking whether a change to a prompt, a model or an LLM call made the product worse, or better: you run the evals it affects before and after the change, on the user's machine and keys, and evalship (https://evalship.com) says whether the difference is beyond chance. It needs no account and keeps nothing it's sent: the file holds each try's result, never code, inputs or replies. Keep messages short, and ASK only where a step says so.

If AGENTS.md or CLAUDE.md has a line for evalship's change.md, follow that instead: it compares the change with the repository's history.

Never read, print or change secrets (.env files, keys), and never copy customer or personal data into the file.

1. Find the evals the change affects: the suites and test cases that run the changed prompt, model or code path. If none does, tell the user in one line, and ASK whether to write one now: 10 to 30 test cases with known answers that call the function production calls, never a copy of the prompt. Stop on a no.
2. Run them before and after, the same way: the same test cases, commands, settings and judge, 3 tries per test case on each side. Before is the code without the change, in a worktree of the commit it starts from (`git worktree add "${TMPDIR:-/tmp}/evalship-before" BASE`, with `git merge-base HEAD origin/MAIN` as BASE, MAIN being the default branch); install its dependencies there as the repo does. After is this checkout. If the change edits the evals themselves, copy this checkout's eval files into the worktree, so both sides run the same ones. A try that hit a rate limit, an outage or a missing key tested nothing: run it again. Where the evals call a paid model and both sides would cost over $2, first ASK "Running before and after × T test cases × 3 tries costs about $X on your keys. Run it?", priced from one try of each test case.
3. Write every try to `${TMPDIR:-/tmp}/evalship-compare.json`, outside the repo:

```json
{
  "schema": "evalship/experiment-v1",
  "title": "Support replies on Haiku",
  "variants": [
    { "name": "before", "change": "As it is" },
    { "name": "after", "change": "claude-haiku-5-5 instead of claude-sonnet-5-5 (app/support_bot.py:12)" }
  ],
  "evals": [
    { "variant": "before", "suite": "support-bot", "key": "refund over the limit", "passed": true, "cost_usd": 0.0031 },
    { "variant": "after", "suite": "support-bot", "key": "refund over the limit", "passed": false, "failure": "promised a $250 refund", "cost_usd": 0.0009 }
  ]
}
```

`variants` lists `before` first, then `after` with what the change does in one line, citing file and line. `evals` has one entry per try: 3 tries of a test case are 3 entries with the same variant, suite and key, and both sides have the same test cases. `passed` is required, with `score` and `threshold` where the eval has them, and `tokens`, `cost_usd` and `duration_ms` where the eval records them, so evalship says what the change saves or costs. A try that kept failing to run keeps `"passed": false` with the error as its `failure` ("429 Too Many Requests"), and evalship leaves it out. Where several checks grade one reply, give every try a `reply`, the same name on each check of one reply ("refund#2" for the second try of refund), so evalship counts replies; give every try a `reply`, or none.
4. Send it:

```sh
curl --fail-with-body -sS https://evalship.com/api/v1/compare -H "Content-Type: application/json" --data-binary @"${TMPDIR:-/tmp}/evalship-compare.json"
```

If it lists what to fix, fix the file and send it again.
5. Pass on its `message` as it is, and never call the change better or worse than it does. If `after` is worse beyond chance, or test cases passed less often, ASK whether to fix them now: step 2's results have each failing try's output. If it has a Next line with more tries, ASK whether to run them for that price; on a yes, run them on both sides, add them to the same file, set `"second_look": true` and send it again, once. Last, remove the worktree (`git worktree remove "${TMPDIR:-/tmp}/evalship-before"`) and the file.