The review
The review is your coding agent's look at your evals, written while it follows the setup prompt. The agent writes it on your machine with your model subscription; evalship checks every citation in it against your repository and shows it as the repository's report.
The review is free for every repository, public or private.
What it contains
| Section | Contents |
|---|---|
| Suites | The inventory: for each suite, its name, framework, path, number of test cases, deterministic and LLM-judge assertion counts, the code it exercises, the models it calls, the command that runs it, its CI triggers, its results format, estimated cost per run, and the line where its pass or fail is decided. |
| Coverage | The folders with evals or tests that the prompt's listing found (candidates, with a digest of the list), and for each folder that isn't a suite, what it holds instead (not_suites). |
| Prompts | Prompt files and inline prompts, with path, line, symbol and the suites that test them. |
| Flaws | Every one that holds up, most valuable first, up to 12. Low-stakes ones are warnings, which Review lists after the rest. |
| Plan | Up to 6 upgrade steps, each marked applied, proposed or skipped. |
| CI | The eval workflow, whether it uploads results, the artifact name, the setup PR URL and the secrets CI needs. |
Each flaw has a kind, a severity (critical, important or warning), a title, one or more pieces of evidence (file, line, an exact quote from that line, and a note), why it matters, and a concrete action. The prompt asks for defects in the evals themselves, each shown with a concrete case, and tells the agent to keep only the flaws a team's lead would call new and useful. One strong flaw is a complete review. A suite CI skips for a missing API key is a flaw too, critical when it's the only one that runs your prompts against a real model: recorded replies can't change when a prompt does. Adding the secret is its fix.
evalship adds one flaw of its own, an eval gap: every prompt and AI call the review lists that no suite exercises. It ranks between critical and important, and you can skip a prompt that isn't worth an eval.
| Kind | Meaning |
|---|---|
not_in_ci |
Evals exist but CI never runs them, or not on pull requests. |
untested_prompt |
A prompt or AI call that no suite exercises. |
untested_instruction |
An explicit requirement in a prompt, such as "never promise refunds above $100", that no test case checks. |
eval_bug |
A defect in how an eval decides pass or fail: the answer is leaked into the prompt, averaged scores hide a failing hard rule, errors or skips count as passes, the scorer sees truncated or wrong input, or correct answers fail. |
too_easy |
Test cases that can't realistically fail, such as checking only for non-empty output, or a threshold of 0. |
judge_flaw |
An LLM judge you can't trust: it grades its own model's output, is swayed by the answer's claims, or uses an undefined scale. |
judge_only |
Every assertion is LLM-as-judge; the flaw names deterministic checks that would catch the same failures cheaper. |
thin_dataset |
Few test cases, near-duplicates, no edge or adversarial cases, or one language or document layout while the product handles several. |
cost |
The most expensive suites, repeated model calls, no caching, or a large judge model where a small one would do. |
duplicated_setup |
Two frameworks testing the same thing, or dead eval files. |
nondeterminism |
Temperature above 0 without repeats or tolerances, live network calls, or no seeds. |
Schema
The review is JSON following evalship.com/schema/audit-v1.json. The required top-level fields are schema ("evalship/audit-v1"), repository (OWNER/REPO), suites and findings. Unknown fields are rejected.
For a connected repository (every setup since prompt v29), the agent pushes the review to a branch named evalship/review on your own GitHub, as its only file, evalship-review.json, and asks evalship to read it:
curl -sL https://evalship.com/api/v1/setup/SESSION/review
evalship reads the branch through its GitHub App, runs the checks below and answers in plain text: sent, with the link to the review, or what to fix. The agent then deletes the branch, or leaves it if your agent's permissions don't allow that: evalship reads only its latest commit. The audit never leaves GitHub on its way to evalship, so Claude Code's auto mode, which often blocks uploads of repository content to other sites, has nothing to block.
When the repository isn't connected, or the agent can't push to it, it posts the review instead:
curl -sS -X POST https://evalship.com/api/v1/audits \
-H 'Content-Type: application/json' \
--data @/tmp/evalship-audit.json
| Response | Meaning |
|---|---|
201 |
Accepted. The body has url (the claim link), expires_at and a message. |
400 |
The body is not valid JSON. |
413 |
The review is larger than 400 KB. |
422 |
The review doesn't match the schema, a candidate folder is covered by no suite and no not_suites entry, or, for a repository the setup is connected to, the default branch has changed eval files or workflows since the audit's commit. details lists what to fix. |
POST /api/v1/audits/check runs the same checks without saving the audit, so the agent can fix it before showing it to you. Strings over a field's length limit are shortened, flaws past the twelfth and plan steps past the sixth are dropped.
The endpoints need no authentication. A posted review without a setup key is attached to a repository only through its claim link.
The claim link
The claim link looks like https://evalship.com/claim/<token>. Opening it signs you in with GitHub, installs the evalship GitHub App on the repository if needed, and attaches the review to the repository. You need push access to the repository on GitHub.
An unclaimed link expires after 7 days; a claimed one redirects to the report. For an expired link, paste the setup prompt into your agent again to get a new one.
Citation checks
Coding agents can get file names, line numbers and quotes wrong. After a review is claimed, evalship fetches every file its flaws cite, at the commit the review's commit names when GitHub has it, otherwise from the default branch and then the setup PR's branch (evalship/setup), and checks each piece of evidence:
- The file exists and the cited line is within it.
- Its quote appears within 3 lines of the cited line, or anywhere in the file when no line is given. Every piece of evidence needs a quote; the API turns evidence without one away. Whitespace, quotes and backticks are ignored, and the first 120 characters of the quote are compared.
A flaw stays visible only if every piece of its evidence holds; the others are hidden and Review says how many. Checks run once, right after the claim, and usually finish within seconds.
The same check covers the review's map of the repository, which the PR comment builds on. A suite's scored_by whose quote isn't on its line is dropped, so a scoring change is then found by the suite's own files. A prompt or an exercises entry whose file doesn't exist is dropped. A symbol the file doesn't contain is dropped from its entry, so the prompt is found by its line and the exercised file counts as a whole. The audit lists what was dropped under unverified.
Every Monday, evalship finds folders on the default branch that the setup prompt's listing (step 1) would now name as candidates by their file names, and that the review neither listed nor covers. The review shows them as new since it ran, and the dashboard and the weekly email ask for another review. Paste the setup prompt again to review them.
Who sees what
| Repository members | Everyone else | |
|---|---|---|
| How many flaws, and the suite inventory | Yes | Public repositories only, unless Public dashboard is turned off in its Settings |
| Flaws and upgrade plan | Yes | No |
| Pass rates and recent runs | Yes | Public repositories only, unless Public dashboard is turned off in its Settings |
Repository members are people with access to the repository on GitHub, read-only included, and people invited to its organization on evalship. Changing its settings, attaching a review and dismissing flaws take push access. Someone invited by email who signs in without GitHub is a reader: they see everything members see, and can't change what takes push access.
After setup
The review keeps working on every pull request:
- Fixed flaws. When a pull request changes a file a flaw cites (or, for a missing test case, a suite's files), evalship counts it fixed when every line it quotes is gone, and otherwise asks the model whether the diff fixes it. The comment says so with ✅. The repository's dashboard then marks the flaw "Being fixed in #N" while that pull request is open, and "Fixed in #N" once it merges. A flaw stays credited to the first merged pull request that fixed it, so a later one, such as a pull request stacked on it, isn't credited or asked about it again. After a merge, the next run on the default branch checks the open flaws the pull request touched once more, so a fix that only works once merged is counted.
- Scoring changes. The review records the line where each suite decides pass or fail (
scored_by). A pull request that removes or rewrites that line gets a ⚖️ note when a result moved, because its before and after results may not compare. - Untested changes. The prompts and suites the review lists tell evalship which AI calls in a pull request no eval exercises. See Untested changes.
See How the PR comment works for how these appear.
Reviewing again
Review shows when the review was written, and by which agent and model. To review again, open the "Review again" tab above the review and paste its line into your agent. Review shows the newest review, and the earlier ones stay a tab away. Pasting the setup line again without it continues setup from where it stands instead, including the upgrades marked proposed.
The reviewed badge stays green for 90 days after the latest review was claimed.
Dismissing a flaw
Someone with push access can dismiss a flaw as Wrong, Won't fix, Fixed or Later, with an optional why, which shows next to it. Later doesn't hold setup up, and Fix flaws lists the flaw until a pull request fixes it. Your coding agent dismisses the ones you agree to the same way. Fixed with a pull request's number ("in #5") counts the flaw as fixed once #5 merges. Fixed with none, such as a secret added to CI, counts as fixed right away. Reopen brings a dismissed finding back.
Fixing a flaw
Each flaw's fix instructions are one line for your coding agent, with a link to a page for that flaw alone. The agent checks it against the default branch, fixes it in a pull request or dismisses it with a reason you agree to, waits for evalship's comment, and tells evalship what it dismissed.