What your agent does

If a web fetch tool gave you a summary of this page, download the whole page and follow it instead, because a summary drops the exact commands and JSON these steps need: `curl -sL 'https://evalship.com/setup.md?key=KEY'`

# Set up evalship for OWNER/REPO (setup prompt v112)

You are setting up evalship (https://evalship.com) in this repository. evalship reads LLM eval results from GitHub Actions and comments on every pull request with what changed and why.

Ask the user only where a step says ASK, and wait for the answer. If you have a tool that asks the user a question with options to pick from (in Claude Code, AskUserQuestion; in Codex, request_user_input where it's available), use it for every ASK, with your recommended option first; otherwise ask in one short message. Keep each question to two lines at most, in plain words ("GitHub Actions", not "workflow"), and put lists, diffs, details and links in a message just before it: a question can't make a link clickable, a message can. Keep messages short.

Before you run anything else, ASK this question word for word, even when the user asked you to set up, since it offers a stronger model first: "The initial review and setting up the PR feedback bot takes ~15 minutes. Then, we'll fix everything the review finds which can take longer. Start now?" Use these options and descriptions word for word, replacing HOW for the agent you are: in Claude Code, Opus with `/effort xhigh`; in Codex, `/model` with its highest reasoning effort; elsewhere, the model picker.
- Start now: "It shows you its plan before it changes anything."
- Switch model first: "The review finds more with your strongest model at high effort (HOW). Stops here; say go once you've switched." If they pick this, stop.
- Not now: "Nothing runs."
If you came here from this setup's sign-up step, you already asked that: skip it.

Once they start, if you have a task list or plan tool, put the four steps in it (find the evals, connect CI, review the evals, send the review) and mark each one done as you finish it. When you finish a step, say so in one line with what's next ("Step 1 of 4 done: found 3 suites. Next: the CI change."). Then work through the steps.

Ground rules:
- Never read, print, create or change secrets (.env files, keys, credentials). If CI needs one, the user adds it (step 2f).
- Never copy customer or personal data from the repo, logs or a database into eval cases, the review or a pull request: write made-up cases instead, in the shape real inputs have (their fields, columns and layout), with only the values made up.
- Never weaken, skip or delete existing evals or assertions, and never change product code.
- Every claim about the repo must cite a file and line you read.
- If a permission or safety check blocks a command, don't work around it: tell the user in one line which step it was for, and give them the command to run themselves (in Claude Code and Codex, by typing `!` before it), and end that message with **Say done once it has run.** in bold, as its last line. Then read the command's output and carry on from there.
- evalship supports GitHub Actions only. Find OWNER/REPO with `git remote get-url origin` (it should be OWNER/REPO).

## Step 1: Find the evals (read only)
Framework names miss suites (a benchmark script, a behave feature in a mostly TypeScript repo, a gate script CI runs), so start with this read-only listing from the repo root:
```sh
# Lists up to 4 matching files per directory, with the directory's total.
per_dir() { awk -F/ '{d=$0; sub(/\/[^\/]*$/, "", d); if (d==$0) d="."; n[d]++; if (n[d]<=4) s[d]=s[d] "\001    " $0} END {for (d in n) print n[d] "\t" d s[d]}' | sort -rn | head -${1:-50} | tr '\001' '\n'; }
evalish=$(git ls-files | grep -iE '(^|/)(evals?|benchmarks?|bench|golden|judges?|graders?|scorers?|rubrics?|accuracy|e2e)[/_.-]|[^/]*(eval|bench|judge|grader|scorer|golden|rubric)[^/]*$|promptfooconfig|(^|/)SKILL\.md$|\.feature$' | grep -viE '\.(png|jpe?g|gif|svg|lock)$|node_modules/|vendor/')
llm=$(git grep -lE 'openai|anthropic|openrouter|@ai-sdk|litellm|langchain|google\.genai|generativeai|bedrock|ollama|deepeval|ragas|promptfoo|braintrust|langsmith|inspect_ai|evalite|vitest-evals|llm_judge|model_graded' -- '*test*' '*spec*' '*eval*' '*bench*' '*judge*' 'scripts/*' '*.feature')
tests=$(git ls-files | grep -vE 'node_modules/|vendor/' | awk -F/ '{ for (i = 1; i < NF; i++) if ($i ~ /^(tests?|spec|__tests__|e2e|e2e-tests|system|features|integration)$/) { print; break } }')
echo "## Files that look like evals, benchmarks, graders, fixtures or skills"
printf '%s\n' "$evalish" | per_dir 50
echo "## Tests and scripts that call an LLM or an eval framework"
printf '%s\n' "$llm" | per_dir 30
echo "## Package scripts, make targets and CI steps that run tests or evals"
git grep -nE '"(test|eval|evals|bench|benchmark|accuracy|e2e)[a-zA-Z:_-]*"[[:space:]]*:' -- '*package.json' | head -40
git grep -nE '^(test|eval|evals|bench|benchmark|accuracy|e2e)[a-zA-Z_-]*:' -- 'Makefile' '*.mk' 'justfile' 'Justfile' | head -20
git grep -nE '(pytest|vitest|jest|behave|promptfoo|deepeval|inspect eval|bt eval|braintrust eval|evalite|npm (run|test)|pnpm|yarn|uv run|python3? [^ ]*(eval|bench|gate|judge)|make )' -- '.github/workflows/*' | grep -v 'install' | head -60
cands=$(printf '%s\n%s\n%s\n' "$evalish" "$llm" "$tests" | grep -v '^\.github/' | awk -F/ 'NF { d = NF == 1 ? $0 : $1; if (NF > 2) d = d "/" $2; if (!(d in seen)) { seen[d]; print d } }')
echo "## candidates (folders with evals or tests, two levels deep; copy both lines into the review as they are)"
printf '%s\n' "$cands" | awk 'NF { out = out (out ? ", " : "") "\"" $0 "\"" } END { print "\"candidates\": [" out "]," }'
printf '"candidates_digest": "%s"\n' "$(printf '%s\n' "$cands" | grep . | { sha256sum 2>/dev/null || shasum -a 256; } | cut -c1-64)"
```
Treat every directory, script and CI step it prints as a possible suite, and list any directory with many matches (`git ls-files <dir>`). Follow what CI and package scripts actually run, too; the listing won't catch everything. Copy its last two lines, `candidates` and `candidates_digest`, into the review exactly as printed (evalship checks the digest, so a shortened list is turned away): every candidate folder must be covered by a suite's `path` or by a `not_suites` entry that says in one line what the folder holds instead (unit tests of product code, docs, UI). evalship checks this and turns the review away if a candidate is missing.
Look for: promptfoo (promptfooconfig.*), Evalite (*.eval.ts), vitest-evals (describeEval), DeepEval (deepeval, assert_test), Braintrust (Eval( from braintrust), LangSmith (evaluate( with langsmith), Inspect AI (@task), OpenAI Evals (evals/registry), Ragas, Opik, Langfuse experiments, agent skill evals (SKILL.md folders with evals/evals.json, promptfoo configs that load a SKILL.md, scripts that run an agent on skill scenarios), benchmark scripts (benchmarks/, *_benchmark.py) and hand-rolled evals (evals/, test/evals/, spec/evals/, tests that call an LLM SDK and assert on its output). For a skill, the SKILL.md is the prompt under test. Every suite you find goes in the review's `suites` list, including benchmark scripts and suites you only skim. The one exception is a benchmark of a model on its own: if no prompt, code or config from this repo is under test (it asks a model fixed questions to see what the model knows or to rank models), put it in `not_suites`, because no pull request here can move its score. Also find the prompts and LLM calls they cover (prompt files and inline prompts next to OpenAI, Anthropic, Google, LiteLLM, LangChain, Vercel AI SDK or similar calls), and which GitHub Actions workflows run the evals. Open the tests that mention the product's model client to see which call a real model.
Evals also hide outside the default branch: on a branch nobody merged, or in a file git doesn't track, such as a gitignored scratch script. List those read-only too:
```sh
base=$(git rev-parse -q --verify origin/HEAD || git rev-parse HEAD)
llm='openai|anthropic|openrouter|@ai-sdk|litellm|langchain|google\.genai|generativeai|bedrock|ollama|deepeval|ragas|promptfoo|braintrust|langsmith|inspect_ai|evalite|vitest-evals|llm_judge|model_graded'
evalish='(^|[/_.-])(evals?|evaluat[a-z]*|bench[a-z]*|judges?|grad(ers?|ing)|scor(es?|ers?|ing)|golden|rubrics?)([/_.-]|$)'
echo "## Branches the default branch hasn't merged, with their eval-like or model-calling files"
git for-each-ref --no-merged="$base" --sort=-committerdate --format='%(refname:short) %(committerdate:short)' refs/heads refs/remotes | head -15 | while read -r b d; do
  f=$(git diff --name-only --diff-filter=AM "$base...$b" 2>/dev/null | grep -vE 'node_modules/|vendor/|\.(png|jpe?g|gif|svg|lock)$')
  [ -n "$f" ] || continue
  hits=$( { printf '%s\n' "$f" | grep -iE "$evalish"; printf '%s\n' "$f" | xargs git grep -lE "$llm" "$b" -- 2>/dev/null | sed "s|^$b:||"; } | sort -u)
  [ -n "$hits" ] && { echo "$b ($d): $(printf '%s\n' "$hits" | wc -l | tr -d ' ') files"; printf '%s\n' "$hits" | head -6 | sed 's/^/    /'; }
done
echo "## Files git doesn't track that call a model (names only)"
{ git ls-files --others --exclude-standard; git ls-files --others --ignored --exclude-standard; } | grep -vE '(^|/)\.|node_modules/|dist/|build/|venv/|site-packages/|__pycache__/|coverage/|vendor/|secret|credential' | grep -E '\.(py|ts|tsx|js|mjs|cjs|rb|ipynb|ya?ml|jsonl?|md)$' | head -3000 | tr '\n' '\0' | xargs -0 grep -lsE "$llm" | head -10
```
None of these is a suite: CI doesn't run it and evalship can't read it, so it stays out of `suites`, `candidates` and the findings, and you open no `.env` or other file that holds secrets. But a test or script there that runs the product's model calls on real inputs is the user's earlier work on evals: read it on its branch (`git show BRANCH:PATH`), name it when you tell the user what you found, and if the default branch has no evals, draw the first suite's cases from its inputs.
Tell the user what you found in at most three lines, plus one for what's outside the default branch. 
Then send the plan for the setup pull request in one message, as text the user sees: what it holds, a few words each, and that nothing merges until they do. If there are no evals, say so and propose a first suite for the LLM call that matters most: the one the product exists for, or the one where a wrong answer costs money, safety or data; name that call (file and line) and the 5 to 10 test cases you'd write, a few words each. If there are evals but no GitHub Actions job runs them yet, propose one that runs them on every pull request that changes what they test and on every push to the default branch. Otherwise it holds what step 2 adds to the workflow that runs them (the results file, the upload, a missing trigger). It also adds an Evals section to the instruction files the repo's coding agents read (step 2e): name them. If it has none, and the LLM calls the evals test are in this repo, say "No AGENTS.md or CLAUDE.md found. The pull request adds an AGENTS.md so your coding agent runs the evals a change affects before it finishes, as it does your tests." If the evals call a paid API, say what a run costs, that setup's own pull requests run them about 20 times, and which secret it needs. When the repo doesn't have that secret yet (`gh secret list` prints names only), say the workflow's triggers depend on where the evals run, which you ask with this question. Then ASK "Go ahead with the setup pull request?", with the options go ahead (recommended, first: step 2 makes it and opens it), go ahead without AGENTS.md (second, only when the plan adds one) and skip it (then skip to step 3; without evals, the review's findings say what should be tested first).
If the evals the setup pull request runs need a secret NAME the repo doesn't have yet, ASK in the same message as the plan's question, or the same dialog if your agent has one, so the user answers both at once, "Run the evals in GitHub Actions automatically, or manually on your machine?", with these options and descriptions word for word, PROVIDER being the service it pays for, in plain words, after "an" when it starts with a vowel: my machine (recommended, first) "Your coding agent runs the evals a change affects before it finishes. Your key stays on your machine." and GitHub Actions "Every pull request's evals run on their own. Needs you to add a PROVIDER key to GitHub." If they skip the setup pull request, this answer changes nothing. For my machine, the setup pull request needs no secret and step 2's goal says how its workflow runs: go on to step 2. For GitHub Actions, tell the user now, so they make the key while you work on step 2: open the page where PROVIDER, the service it pays for, makes API keys in their browser if you know it (`open` on macOS, `xdg-open` on Linux, and send the link too), suggest naming the key REPO-github-ci with a spending limit if PROVIDER offers one, of at least 20 runs' cost, since a key that runs out stops every eval, and have them save it in their own terminal with `gh secret set NAME -R OWNER/REPO`, which asks for the value, never in this chat. Making and saving the key is theirs, even when they ask you to: never do either, browser tools included, since its value would pass through this chat; when they bring the key up, check `gh secret list` first. If PROVIDER is Anthropic, add that Claude Max, Ultra and Team plans include $100 to $500 a month of API credit, which a key made in the Console organization they link in claude.ai under Settings > Billing spends first. Don't wait for it: go on to step 2.

## Step 2: Connect CI
Goal: the workflow that runs the evals triggers on `pull_request` and on `push` to the default branch, writes a results file, and uploads it as an artifact named `evalship-results`. No evalship action, token or secret is needed. When the user picked my machine in step 1, it runs the evals only on `workflow_dispatch:`, which c below doesn't change: a workflow you create has no other trigger, `paths` filter or fork condition, and in one that runs more than the evals, the eval job gets `if: github.event_name == 'workflow_dispatch'`. Their coding agent follows it to run the evals, and a pull request's comment gives it the line for that pull request.
First suite, only if the plan in step 1 had one and the user went ahead (write it first; a to f then apply to it):
- Call the function production calls, with its prompt, model, tools and output parsing, against a real model; never copy the prompt into the test. Fake only what's around it (database, other services), with fixtures that don't hint at the answer. If it can't run without a server, test the narrowest function that builds the prompt and calls the model, and say in the pull request what that leaves out.
- 5 to 10 test cases, each its own test with a stable name: the request the product gets most, an input that tempts the model to break each rule in the prompt where a mistake costs money, safety or data, and one it should decline if the prompt says when to. Take inputs from the repo first (README examples, fixtures, seed data, docs, issues) and write the rest, varying phrasing, length and missing details, not near-duplicates. Where the product reads documents or structured input, cover each layout, field set and file type its parsing code handles: a made-up file with a column real ones lack can pass while every real one fails.
- Check in code first: parsed fields, the tool called and its arguments or the state it changed (not what the reply says it did), a number, name or date as a whole word, and what the reply must not claim. A judge only for what code can't check: one pass/fail question per failure mode with written Pass and Fail definitions, reason before verdict, a different model than the one under test from the same provider (so CI needs no other key), and no clear verdict counting as fail. No 1-to-10 or similarity scores.
- Use the repo's test runner, in a folder, marker or config its usual test command skips, so unit tests don't call the model; add a dependency only if nothing installed can call it.
- Before running the model, try each check on made-up replies: the right answer reworded and with a caveat must pass; empty, the input echoed, a negation, a near miss, a refusal and a claimed action the product can't take must fail. Fix checks until they do, and commit those replies as tests of the checks: they call no model, so keep them out of the eval run and its results file and give them a CI step of their own before the evals, without continue-on-error, so a later change to a check is tested without the tests counting as evals. Then have a subagent with no memory of writing it (in Claude Code, the Agent tool) review the suite: give it only the suite's files, the product code it calls and the numbered rules on step 3's page (`curl -sL 'https://evalship.com/setup.md?s=SESSION&step=3'`), copied in full, not why you wrote it as you did; if you can't start one, reread the suite cold against those rules. Fix what holds. Then run the suite once: if the environment lacks the model's key (never read or print it), check that it fails only for the key and still writes the results file. A case failing a right answer is a wrong check: fix it; one failing because the product is wrong stays, as the suite's first catch. A case that fails in some of the setup pull request's runs and passes in others is a bug users hit that often, but evalship sets it aside as flaky and stops checking it. Where the product is right only most of the time on any input (a reviewer's notes, a classifier's labels), score those answers as one case, the share that's wrong, failing only on more than chance gives in 1 run of 20 at the rate those runs measured, so the default branch passes and a change that makes it worse fails, and say the rate in the pull request. With the diff, say how both came out ("4 right replies pass, 9 wrong fail; 7 of 8 test cases pass, the refund one promises $250 where the prompt caps it at $100").
a. Results file: JUnit XML (pytest `--junitxml=eval-results.xml -o junit_family=xunit1`, vitest `--reporter=junit --outputFile=eval-results.xml`, jest-junit, rspec_junit_formatter, go-junit-report, Maven's or Gradle's own reports with their whole folder uploaded (one file per test class), PHPUnit `--log-junit eval-results.xml`, cargo-nextest with a `[profile.ci.junit]` path since `cargo test` can't write it on stable), promptfoo JSON (`promptfoo eval -o eval-results.json`) or evalship JSON (https://evalship.com/schema/v1.json). In pytest, add `record_property("score", s)` and `record_property("threshold", t)` where an eval computes them, and `record_property("input", ...)` and `record_property("output", ...)` with the case's input and the model's reply, so the report can show what the model said before and after a change (other runners: write evalship JSON, or print the reply into the JUnit `system-out`). If evals report to Langfuse, Braintrust, LangSmith or a similar platform, keep that and have the same script also write evalship JSON from the results it gets back, with each case's `url` set to its trace or experiment link, so the pull request comment links to it (examples: https://evalship.com/docs/platforms).
b. Right after the eval step:
```yaml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml
```
c. If a workflow already runs the evals, keep it as it is and add only what's missing (the results file, the upload step, a missing trigger). Its results must reach evalship on pull requests and on pushes to the default branch, also when evals fail: an upload that runs only on success, a default-branch run that is manual or weekly only, or results kept only in a job summary leave evalship nothing to compare against, so fix those too. If the code the evals run asks for a discounted tier that turns calls away when the provider is busy (Gemini's or OpenAI's `service_tier: "flex"`), have eval runs ask for the standard one, through a variable the workflow sets: the replies are the same, and a turned-away call stops an eval with a 429 instead of testing it. If no workflow runs them, create `.github/workflows/evals.yml` if the plan in step 1 had it, or a first suite, and give its eval step `continue-on-error: true`: the job then stays green when an eval fails or flips, and evalship's check, which compares with the default branch and ignores flaky evals, decides whether a pull request broke them. A `pull_request` trigger you write, in a new workflow or added to an existing one, gets a `paths` filter, so a pull request that can't change the results doesn't pay for a run: the suites' own files (cases, datasets, scorers, judge prompts, eval config), the product code and prompts in each suite's `exercises` and `prompts`, the code they call to build the model's input or read its reply (retrieval, context, tools, output parsing), the files that pick the model or its settings, the dependency manifest and lockfile that pin the model SDK, and the workflow file itself. Use folder globs where that code has a folder of its own, so a new file there is covered, and the whole folder when you're unsure: a path the filter misses skips the evals on a pull request that changes it. An existing `paths` filter keeps its paths and gets those it misses from this list. List the filter's paths in the pull request description, with two notes: deleting `paths` runs the evals on every pull request, and a branch rule should require the `evalship` check, not the eval job, which waits forever on a pull request the filter skips. Leave `push` to the default branch without one, so every merge keeps the baseline current; a pull request the filter skips that changes a prompt or LLM call still gets evalship's comment. Add `workflow_dispatch:` to a workflow you create or change, so the user can run the evals on a branch whenever they choose (`gh workflow run`). Run only what a pull request can change: a comparison arm that loads none of the repo's prompts or code (a no-prompt baseline, another product's prompt) gives the same results on every pull request and may fail by design, so leave it out of CI (promptfoo `--filter-prompts`, a pytest marker, a separate config) and keep it in the repo for its own benchmark, saying so in the pull request description. If one run takes more than about 5 minutes (time it while you test the results file), it's usually waiting on the model one case at a time, as test runners run them by default: run cases that share no state concurrently in the same job (pytest-xdist `-n 4`, promptfoo `--max-concurrency`, a case's repeats with `Promise.all` or `asyncio.gather`), up to the provider's rate limit and any limiter of the product's the eval calls through. Not vitest's concurrent tests (`--sequence.concurrent`, `test.concurrent`): they put a test's printed reply under another case in the JUnit report, so split vitest's cases across parallel jobs as below. If it still takes more than 5 minutes and the eval runner can run part of the cases (a file list, `pytest -k`, `promptfoo --filter-pattern`, a fixture argument), split it across up to 4 parallel jobs with a `strategy.matrix`, each uploading its own artifact, `evalship-results-${{ matrix.shard }}` (evalship merges them): a 16-minute job becomes about 4. Split by case, not by file: vitest's and jest's `--shard` split by file, so one big file keeps the run waiting on its job (have the eval take every Nth case by its index instead), and check with the runner's list command (`vitest list`, `pytest --collect-only -q`) that the jobs together list every case once: vitest fails a `describe` left with no cases. Say how long a run took and how long it takes now in one line when you show the diff. In either case: if the workflow already has `cancel-in-progress: true`, change it to `cancel-in-progress: ${{ github.event_name == 'pull_request' }}`, so every push to the default branch keeps its results (never add a `concurrency` group, and leave `false` as it is); and if the evals need a secret, the job skips pull requests from forks and Dependabot, which don't get it and would fail every eval: `if: github.event_name != 'pull_request' || (github.event.pull_request.head.repo.full_name == github.repository && github.event.pull_request.user.login != 'dependabot[bot]')`.
d. Unless the repo already has one, add `.evalship.yml` at its root so a pull request that breaks an eval that passed on the default branch fails evalship's check. Add it even when the evals are noisy: evalship ignores evals it has seen flip, a new suite's failure counts only when it repeats, and the check blocks merging only once the team makes it required.
```yaml
gate:
  fail_on_new_failures: true
```
e. Coding agents check a prompt change by trying a few inputs, and those checks are gone when the chat ends. To keep them as test cases, add this section, filled in for the suites (a line each when there are several), to the end of each instruction file the repo's coding agents already read that has no Evals section yet, found with `git ls-files`: AGENTS.md, CLAUDE.md or GEMINI.md at the root, in `.claude/` or in a folder above the LLM calls, `.github/copilot-instructions.md`, `.cursorrules`, `.windsurfrules`, or a new `evals.mdc` in `.cursor/rules/` (only the imported one when one imports another). If it has none, create AGENTS.md with the section and a CLAUDE.md holding only `@AGENTS.md`, since Claude Code reads CLAUDE.md only, unless the user picked go ahead without AGENTS.md in step 1. If the prompts and LLM calls the suites test live in another repository, add nothing here: agents change them there, so send the user the section to paste into that repository's agent file if it has one, with SUITE_PATH named as this repository's and without its last sentence, since change.md finds the change in this one. Change nothing else in them.
```markdown
## Evals
When you change a prompt, a model or an LLM call, add or update a test case for the change in SUITE_PATH and try it with `COMMAND`. Before you finish, run the evals the change affects, as you would its tests: follow https://evalship.com/change.md?repo=OWNER/REPO, which sends evalship the change's diff with its results.
```
f. Show the diff. Unless the user picked my machine in step 1, if the evals need secrets in CI that the repo still doesn't have (`gh secret list`), ASK, with NAME the variable the eval code reads and PROVIDER the service it pays for in plain words: "Your evals call PROVIDER, the model API your app uses, so GitHub Actions needs your PROVIDER API key, saved as the secret NAME. Add it before the pull request runs?", with the options added it (then check with `gh secret list`) and skip (then say its first eval run will fail for lack of it). Tell them to add it in their own terminal with `gh secret set NAME -R OWNER/REPO`, which asks for the value, or at https://github.com/OWNER/REPO/settings/secrets/actions/new, never in this chat, so the value stays out of it. Then open the setup pull request, as the plan said: commit on a branch named `evalship/setup`, run `gh pr create`, and send its link. Change nothing else.

## Step 3: Review the evals (read only)
The team knows their own repo, so the review is only worth reading if it tells them something they don't know. That is rarely which checks are missing. It is usually a defect in the evals themselves, which only shows when you read the eval code: the scorers, judges, thresholds, harness and datasets, not just the list of cases.

First, list every suite from step 1 with the files that decide its result, and note in `scored_by` the line where its pass or fail is decided (the assertion, threshold or verdict), quoted exactly; never a README or docs line, and for a config-driven runner (promptfoo, Braintrust) the assertion or threshold line in its config: the scorer or grader code, the judge prompt, the pass threshold and how scores are combined, the mocks and fixtures, the dataset it loads, and any README, docstring or comment that explains its thresholds, skips or mocks. Open every file on that list, and as you start each suite tell the user in one line which one and how far along you are ("Reviewing suite 2 of 5: support-bot"), so they can follow along. Don't sample or skim: the defects worth reporting are usually in the suites you would have skipped. Then follow each suite into the product code it claims to test. List the instructions of each product prompt it tests (its rules, limits, and what the answer must keep, include or never say) and next to each the check that would fail an answer breaking it, or none. Compare the suite with what production sends the model: the same function or a copy of it, the same system prompt, tools and context, the same message shape and output parsing. Also read how that product code fails: a call that returns None, an empty list or a default on an exception makes every case that expects "nothing" pass during an outage. Then look for these, in this order:

1. Evals that can't fail or measure the wrong thing (eval_bug, too_easy). For each scorer, try these outputs on it, in every case: an empty or boilerplate answer, the input echoed back, a negation ("I can't offer a refund" passes a check for "refund"), a refusal, a near miss (a wrong number, date or name that contains the right one: "12 hours" passes a check for "2 hours"), a reply that claims an action the product can't take (it has no tool for it), and one fixed answer when every case expects the same outcome (always refuse, always call the tool); and a right answer with a caveat or an extra detail, which should pass. Where the scorer runs without a model, run it on them in a scratch script outside the repo that imports it; otherwise work them out from its code. Also look for:
   - the case's prompt, fixture, system prompt or few-shot examples state the answer or rule the scorer then checks for; a case whose input doesn't trigger what its name or rubric says it tests
   - a mock, stub or tool schema enforces the behavior under test, or its output (an error message, a hint) steers the model to the right answer, so the check tests the mock, not the model
   - scores averaged across graders or cases, so a hard rule (a leaked secret, a safety rule) can fail while the case passes (a promptfoo `threshold` or `weight: 0` can do this); thresholds low enough that everything passes; scores logged but never asserted or reported anywhere, so a drop goes unnoticed
   - errors, timeouts, skips or unparseable or "unknown" judge verdicts counted as passes or dropped from the denominator (Ragas turns failed rows into NaN and leaves them out of the mean); judge replies parsed by substring (`"pass" in reply` matches "does not pass"); a suite that skips itself when a key or service is missing (for example on pull requests from forks) while CI still goes green
   - the scorer or judge sees truncated input (`text[:4000]`), the wrong field, or a cached output that doesn't change with the prompt or model
   - a checked-in dataset the eval never loads, cases silently filtered out, or the eval calling a different prompt, model, temperature or code path than production (an eval at temperature 0 for a product at 0.7 passes on replies users never get)
   - agent evals that grade what the agent says (its final message, or a judge reading the transcript) instead of the resulting state (file, database row, API call); that check the expected change but not that nothing else changed; that share state between trials; or where the agent can read the expected answers or edit what the grader reads (the eval folder, tests, .git history, the network)
   - for agent skills: a with-skill and without-skill comparison where the two runs differ in more than the skill (a tool, a prompt, a fixture only one side gets); trigger evals whose negatives are far from the skill's boundary, so routing can't really fail
   - for RAG: the eval feeds fixed or gold passages instead of what the retriever returns, so retriever changes can't fail it; faithfulness or groundedness is the only answer check, though it scores agreement with the retrieved passages, not correctness; retrieval questions generated from the chunk they must find, reusing its wording
2. Evals that disagree with the product (eval_bug): check each rubric, judge prompt, expected answer and threshold against the product code it grades: a rubric that names a tool, field or step that doesn't exist; a threshold one off from the one in the prompt (`< 4` against `< 3`); a check computed and never used in the verdict; cases that share state, an account or a container between runs, or create real resources without cleaning up. Evals that fail correct answers: an expected answer that is wrong or contradicts the current prompt; a check so narrow that a valid answer fails (one accepted wording, an exact float, a tool-call order or argument that doesn't matter, a file or format the case never asked for). Teams learn to rerun or ignore red.
3. Judges you can't trust (judge_flaw): the model under test grades its own output, including by default (Inspect's `model_graded_qa` with no grader model, a promptfoo default provider that also grades); the judge reads the candidate's own reasoning or claim of success and is swayed by it; an open-ended answer graded on matching a reference's wording (a reference for a factual check is good practice); a vague or unanchored scale (1 to 10 without definitions) instead of pass/fail per failure mode; one judge call asked to catch everything at once; pairwise judging without swapping order.
4. Rules no case checks (untested_instruction), for the instructions that matter most (safety, money, legal, data loss, the product's core promise), and prompts or LLM calls no eval exercises (untested_prompt). Take the product prompt's instructions one at a time and find the check that would fail an output breaking each. One the cases check only in part is untested too: the prompt says to cite the source document, and the cases check the answer but never the citation. Read the recorded or logged outputs against the same instructions: one that already breaks a rule while its case passes proves it.
5. Only when specific and material: thin_dataset (a failure mode the prompt handles that no case covers, near-duplicate cases, every case in one document layout or field set while the parsing code handles several), nondeterminism (sampling without repeats where results flip), judge_only (only if you can name the deterministic checks that would catch the same failures).

Some teams run evals to monitor, not gate: scores are tracked over time or posted on the pull request, and the build doesn't fail on them. When the repo says so, that's a valid choice, not a finding. Review what the monitoring reports instead: a score that can't move, a model or dataset that silently drops out of the run, results that never reach the report.

Write every finding that passes the checks below, most valuable first: a defect that lets a pull request go green when it should fail ranks above one that skews a benchmark someone runs by hand. Don't pad, and give each defect one finding: two findings with the same fix are one. For each one:
- Show that it's real: name one input, case or run that passes when it shouldn't (or fails when it shouldn't), and why. For each eval workflow, look at its last few runs (`gh run list --workflow <file> -L 5`, then `gh run view <id>`): a step skipped or failing on every run (a missing secret, `if: false`, a broken command) means the suite isn't really running, which is a finding when the repo doesn't say so. A suite skipped for a missing secret is one even when the repo or a pull request says it needs the key, and critical when no other suite in CI runs the prompts against a real model, since recorded or replayed replies can't change when a prompt or model does: the user adding the secret is its fix, not an upgrade. If the repo keeps recorded results, snapshots or transcripts, or CI has run the suite (`gh run view <id> --log`), read some: a recorded pass on a wrong output, a skip count, or a judge reason that contradicts its verdict is the strongest proof, and a case that fails in every recorded run is more often broken than hard.
- When you find a defect, grep the whole repo for its signature (the helper, slice, default, threshold or copied scorer), not just the suites you listed, and report every place it occurs in one finding.
- Look for evidence against it before you write it: an existing case, test or CI job that covers it, and a README, docs page, TODO or code comment near the cited lines that names it (grep for its key words). If one does, drop the finding. If the repo documents the behavior as a deliberate choice, drop it unless you can show a consequence the repo doesn't mention.
- Skip anything the team's lead would call obvious. Don't report CI gaps (step 2 fixes those; put what's left in the plan), and don't give generic advice ("use a smaller judge model", "add more cases", "validate your judge") unless you tie it to a concrete failure in this repo.
- Grade each finding the way the team's lead would before you keep it: new and useful, true but low stakes, or wrong. Drop the wrong ones: a wrong finding costs more than a missing one. Give a low-stakes one severity warning and put it after the others; evalship shows warnings less prominently.
- Count every number you state (cases, graders, lines) with a command (`grep -c`, `jq length`, `wc -l`), not by eye. Cite files and lines, copy each quote exactly from one line of the file (don't reformat or join lines), and end with one concrete action. When a quote doesn't show the flaw on its own, give it a `note` saying what's wrong there, in a few words ("No eval tests a refund request above $100"). Never quote customer or personal data (names, emails, account details, real conversations): cite a line without it, such as the case's key, and describe the rest in the `note`.

A first suite you wrote in step 2 goes in `suites` like the others, with its files on the setup branch, and the prompts it runs name it in `tested_by`; leave it out of the findings.
A prompt or LLM call no eval tests (rule 4) goes in `prompts` with an empty `tested_by`, never in the findings: evalship makes that finding from `prompts`.
Finding kinds for the JSON: eval_bug, too_easy, judge_flaw, untested_instruction, thin_dataset, nondeterminism, cost, duplicated_setup, judge_only, not_in_ci.
Then write the plan: what step 2 already did, and up to 4 more upgrade steps, each tied to something you saw in this repo, and each about the evals or the job that runs them: CI for the rest of the test suite or a coverage gate is not an upgrade. Don't repeat a finding's action: the review shows those already. Anything without which a regression would pass, such as a suite skipped for a missing secret, is a finding, not an upgrade. Draw them from what good eval suites do: run on every pull request that changes what they test and fail loudly when they can't run; finish within a few minutes, so an eval job that takes longer on pull requests (`gh run view` shows how long) runs its cases concurrently, then split evenly across parallel jobs, as step 2 says; deterministic checks before judges; one binary pass/fail judge per failure mode, with a written reason; judges checked against a few human-labeled examples; hard rules checked on their own rather than averaged; new cases taken from real failures; cases that flipped in recorded runs run 3 or more times with the same name and input (promptfoo `--repeat`), so a drop can be told from chance, only those cases and saying what the repeats cost and add to a run's minutes; stable case names. Mark what step 2 already did as "applied", the rest as "proposed". Change nothing in this step.

## Step 4: Send the review
Write it as JSON to a temporary file outside the repo, following https://evalship.com/schema/audit-v1.json:
```json
{
  "schema": "evalship/audit-v1", "repository": "OWNER/REPO", "agent": "claude-code", "prompt_version": "112",
  "commit": "9fceb02d0ae598e95dc970b74767f19372d61af8", "model": "claude-opus-5-5",
  "suites": [{ "name": "support-bot", "framework": "pytest", "path": "tests/evals/test_support_bot.py", "cases": 12,
    "deterministic_assertions": 12, "judge_assertions": 0, "exercises": ["app/support_bot.py#answer"], "calls_model": true, "models": ["gpt-4o-mini"],
    "command": "pytest tests/evals", "ci_triggers": ["pull_request", "push"], "results_format": "junit", "est_cost_per_run_usd": 0.02,
    "scored_by": { "file": "tests/evals/test_support_bot.py", "line": 41, "quote": "assert score >= 0.8" } }],
  "candidates": ["tests/evals", "app/support_bot"],
  "candidates_digest": "8f764de695187898bdee7fd66f0f73fdb56be92f4eea21cd30a962e34d927ed7",
  "not_suites": [{ "path": "app/support_bot", "reason": "Unit tests of the reply formatter; no model call." }],
  "prompts": [{ "path": "app/support_bot.py", "line": 3, "symbol": "SYSTEM_PROMPT", "tested_by": ["support-bot"] }],
  "findings": [{ "kind": "untested_instruction", "severity": "important", "title": "Nothing checks the refund limit",
    "evidence": [{ "file": "app/support_bot.py", "line": 4, "quote": "Never promise refunds above $100", "note": "no case asks for a large refund" }],
    "why_it_matters": "One sentence.", "action": "One concrete step." }],
  "plan": [{ "title": "Run evals on pull requests", "detail": "One sentence.", "status": "applied" }],
  "ci": { "workflow": ".github/workflows/evals.yml", "uploads_results": true, "artifact_name": "evalship-results", "setup_pr_url": "https://github.com/OWNER/REPO/pull/1", "secrets_needed": [], "runs_locally": false },
  "badge_added": false,
  "progress": { "start": "start", "plan": "go ahead", "pr": "opened", "blocked": "" }
}
```
`commit` is what `git rev-parse HEAD` printed while you read the repo, so evalship checks your citations against the same code; `ci.runs_locally` is true when the user picked my machine in step 1, so evalship waits for their coding agent's results instead of CI's, and then the suites the workflow runs get `ci_triggers` ["pull_request", "push"], since the agent runs them for each pull request and the default branch; `progress` holds the option the user picked at each question so far, a word each (start, plan if you asked, and pr: opened or skipped), and in `blocked` the steps a safety check blocked, comma-separated; `agent` is the agent you are (claude-code, codex, cursor or another short name); `model` is the model you run on, if you know it (leave it out otherwise). Severity is critical when a broken product would still pass (a suite that can't fail, or a safety, money or data rule that can fail unnoticed), important when a real regression could slip through, and warning for the rest; plan status is applied, proposed or skipped; `framework` is a short name (promptfoo, pytest, Inspect AI); `ci_triggers` says how the suite runs once the setup pull request is merged, using pull_request (runs on every pull request, or on those touching its paths), labeled (pull requests only with a label), push, schedule or manual; `results_format` is junit, promptfoo, evalship, another format's name, or none when the suite writes no results file; quotes are copied exactly from the cited line. evalship uses `prompts` and `exercises` to tell which prompt changes in a pull request no eval covers, so make them complete: `prompts` lists every prompt the product sends a model, each where its text starts (prompt files, prompt strings and templates in code, the system prompt of the main answer path, MCP server instructions and tool descriptions, a skill's SKILL.md and each reference file it loads), with `tested_by` naming the suites that run it and check its reply, or [] when none do: a suite that passes whatever the model answers, such as one that only checks the call doesn't raise, doesn't test it. `exercises` lists the product files or `file#symbol` each suite runs (a skill as `skills/<name>/**`), not eval files. `calls_model` is false for a suite that never sends anything to a model (snapshots, schema checks, unit tests of prompt text or of code around the call), so evalship doesn't count it as testing a prompt's behavior; `models` lists every model a suite calls, including judges. Show the user the findings, one line each with its severity, then send the review: they agreed to it when they started. Put the review on GitHub, where evalship reads it: commit the JSON as the only file `evalship-review.json` of its own commit, so the working tree and the current branch stay as they are, and push it to origin as the branch `evalship/review`, on top of what's there, so no force is needed:
```sh
blob=$(git hash-object -w /tmp/evalship-audit.json)
tree=$(printf '100644 blob %s\tevalship-review.json\n' "$blob" | git mktree)
if git fetch -q origin evalship/review 2>/dev/null; then commit=$(git commit-tree "$tree" -p FETCH_HEAD -m 'evalship review'); else commit=$(git commit-tree "$tree" -m 'evalship review'); fi
git push origin "$commit":refs/heads/evalship/review
```
Then run `curl -sL 'https://evalship.com/api/v1/setup/SESSION/review'`: evalship reads the branch, checks the review and says it's sent, with the link, or lists what to fix. Fix the JSON, push it again and run that again until it says sent, and tell the user if a fix changed a finding. Then delete the branch with `git push origin --delete evalship/review`; if that's blocked, leave it and don't ask the user to: evalship reads only its latest commit, and the next review goes on top. If the push fails, for lack of write access, send the review directly instead: `curl -sS -X POST 'https://evalship.com/api/v1/audits?key=KEY&s=SESSION' -H 'Content-Type: application/json' --data @/tmp/evalship-audit.json`.



## After sending: one step at a time
Keep notes about the changes (costs, settings CI copies, what runs where) in the pull request descriptions, not in the chat.
1. Say evalship's "Sent:" line as it is, with how many flaws it found and the link, so the chat says what the review page says, and open the link in their browser too if you can (`open` on macOS, `xdg-open` on Linux) so they see the flaws right away; never describe evalship's check of the review as findings ("it passed", "no new findings"). Once the setup pull request's CI has run, check with `gh pr view NUMBER --comments` that evalship commented on it and say so in one line; when the user picked my machine, the next page runs its evals instead.
2. Then download where setup stands now: `curl -sL 'https://evalship.com/setup.md?s=SESSION'`, and follow it from its step 1, skipping what you already said. It picks up from here, as the review page does: the setup pull request, the review's flaws (fixed or, with the user's say-so, dismissed), the upgrades, the required check and the finish.