The audit
The audit is the review of your evals that your coding agent writes while following the setup prompt. The agent writes it on your machine with your model subscription; evalship checks every citation in it against your repository, stores it, and shows it as the report for the repository.
The audit is free for every repository, public or private.
What it contains
| Section | Contents |
|---|---|
| Headline | One or two sentences on the state of the repository's evals. |
| Suites | The inventory: for each suite, its name, framework, path, number of cases, deterministic and LLM-judge assertion counts, the code it exercises, the models it calls, the command that runs it, its CI triggers, its results format and estimated cost per run. |
| Prompts | Prompt files and inline prompts, with path, line, symbol and the suites that test them. |
| Findings | At most 8, most valuable first. |
| Plan | Up to 6 upgrade steps, each marked applied, proposed or skipped. |
| CI | The eval workflow, whether it uploads results, the artifact name, the setup PR URL and the secrets CI needs. |
Each finding has a kind, a severity (info, notice or important), a title, one or more pieces of evidence (file, line, an exact quote from that line, and a note), why it matters, and a concrete action. The prompt tells the agent that generic advice is not a finding, and to report 2 findings if it finds only 2 real issues.
| Kind | Meaning |
|---|---|
not_in_ci |
Evals exist but CI never runs them, or not on pull requests. |
untested_prompt |
A prompt or LLM call that no suite exercises. |
untested_instruction |
An explicit requirement in a prompt, such as "never promise refunds above 100 euros", that no case checks. |
judge_only |
Every assertion is LLM-as-judge; the finding names deterministic checks that would catch the same failures cheaper. |
too_easy |
Cases that can't realistically fail, such as checking only for non-empty output, or a threshold of 0. |
thin_dataset |
Few cases, near-duplicates, no edge or adversarial cases, or one language while the prompt handles many. |
cost |
The most expensive suites, repeated model calls, no caching, or a large judge model where a small one would do. |
duplicated_setup |
Two frameworks testing the same thing, or dead eval files. |
nondeterminism |
Temperature above 0 without repeats or tolerances, live network calls, or no seeds. |
Schema
The audit is JSON following evalship.com/schema/audit-v1.json. The required top-level fields are schema ("evalship/audit-v1"), repository (OWNER/REPO), headline, suites and findings. Unknown fields are rejected.
The agent sends it with:
curl -sS -X POST https://evalship.com/api/v1/audits \
-H 'Content-Type: application/json' \
--data @/tmp/evalship-audit.json
| Response | Meaning |
|---|---|
201 |
Accepted. The body has url (the claim link), expires_at and a message. |
400 |
The body is not valid JSON. |
413 |
The audit is larger than 400 KB. |
422 |
The audit doesn't match the schema. details lists up to 5 errors; the agent fixes the JSON and retries. |
The endpoint needs no authentication. An audit is attached to a repository only through its claim link.
The claim link
The link from the response looks like https://evalship.com/claim/<token>. Opening it signs you in with GitHub, installs the evalship GitHub App on the repository if needed, and attaches the audit to the repository. You need access to the repository on GitHub.
The link expires after 7 days if it is not claimed. After it's claimed, it redirects to the report. For an expired link, paste the setup prompt into your agent again to get a new one.
Citation checks
Coding agents can get file names, line numbers and quotes wrong. After an audit is claimed, evalship fetches every file its findings cite from the repository's default branch and checks each piece of evidence:
- The file exists and the cited line is within it.
- If there is a quote, it appears within 3 lines of the cited line, or anywhere in the file when no line is given. Whitespace, quotes and backticks are ignored, and the first 120 characters of the quote are compared.
A finding stays visible only if every piece of its evidence holds; the others are hidden and the report says how many. Checks run once, right after the claim, and usually finish within seconds.
Because the check uses the default branch, a finding that cites a file only present in the unmerged setup PR is hidden. To have those findings checked, re-audit after merging.
Who sees what
| Repository members | Everyone else | |
|---|---|---|
| Headline and suite inventory | Yes | Public repositories only, unless the public report is turned off in settings |
| Findings and upgrade plan | Yes | No |
| Pass rates and recent runs | Yes | Public repositories only, unless the public report is turned off in settings |
Repository members are people with access to the repository on GitHub.
Re-auditing
The report shows when the audit was written, by which agent and with which prompt version. To audit again, use "Copy the prompt to re-audit" on the report page (or copy it from evalship.com), paste it into your agent, and open the new link. The newest claimed audit becomes the report. To apply steps marked proposed, paste the prompt again and ask the agent to apply them.
The reviewed badge stays green for 90 days after the latest audit was claimed.
Feedback
Each finding has thumbs up and down for repository members. evalship uses the votes to measure which kinds of findings are useful, and can turn off a kind of finding for all repositories when it is not useful. Reactions on the PR comment are recorded the same way for its bullets.