Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

Quickstart

evalship works in two parts. Setup reviews your evals and shows what they miss, with a fix for each flaw. After that, evalship comments on every pull request that changes your prompts, AI calls or evals, with the evals it really broke, the change that likely broke them, and anything that weakens how your evals are scored.

It works with the evals you already have, whatever runs them, even a handful of asserts in a test file. It does not run your evals: your GitHub Actions workflow runs them and uploads a results file, and evalship reads that file when the workflow finishes. To keep your keys off GitHub, your coding agent runs them on your machine and sends the file instead.

Setup is a one-line prompt you paste into a coding agent inside your repository. The agent shows you its plan before it changes anything, and nothing merges until you do: What your agent does lists every step.

Requirements

  • A GitHub repository with evals, or with AI calls you want evals for. With no evals yet, the agent offers to write a first small suite for your most important AI call.
  • GitHub Actions. Other CI systems are not supported yet.
  • A coding agent that can read the repository and run shell commands, such as Claude Code, Codex, Cursor or GitHub Copilot. The agent uses git, gh and curl. Run it with the most capable model you have at high effort: in Claude Code, that's Opus with /effort xhigh. In our tests on public repositories, that found about a third more flaws worth fixing than Sonnet at its default effort, for a few dollars and about 15 minutes per repository.

Public repositories are free, and so is one private repository with one user; see Pricing and billing.

1. Choose where to start

Start from How it goes
The homepage Copy the one-liner on evalship.com. Your agent starts by giving you a link to sign up and connect the repository.
The guided setup Click "Get started" to open evalship.com/start. It offers the same one-liner, or signs you up with GitHub to go step by step: 1 Sign up, 2 Install the App, 3 Pick a repo, 4 Set up. The repository is connected before the agent starts.

Both install the evalship GitHub App, which GitHub lists as evalship-app (GitHub reserves the name "evalship"). The check it adds to commits is called evalship. The permissions it asks for are listed in Data and privacy.

2. Paste the one-liner into your agent

Start your agent in the repository and paste:

Set up https://evalship.com/setup.md

In the guided setup, step 4 gives you this line with a setup key for the repository you picked, so the review attaches to that repository directly.

The agent fetches the page and follows the instructions on it. Without a key, evalship.com/setup.md (/prompt redirects there) has only the welcome and the sign-up step; the agent downloads the rest once the repository is connected. The title shows the prompt version, and the review records it.

3. Work through the steps with the agent

The agent asks before it starts, then gives you a link to sign up and connect the repository: sign in with GitHub, install evalship's GitHub App on it and click Connect. The link works for an hour; after that, paste the one-liner again. Then it works through the steps, one question at a time, with its question dialog where it has one (Claude Code, and Codex in Plan mode).

Step What the agent does Asks you
1. Find the evals Read only. Starts from evalship's list of the repository's eval and test folders, then looks for promptfoo, Evalite, vitest-evals, DeepEval, Braintrust, LangSmith, Inspect AI, OpenAI Evals, Ragas, Opik, Langfuse experiments and hand-rolled evals, the prompts and AI calls they cover, and the workflows that run them. Earlier eval work on branches you never merged or in untracked scripts is named too, as a source of cases. Once, with its plan for the setup pull request: a GitHub Actions job when none runs your evals, or a first suite when there are none
2. Connect CI If you asked for a first suite, writes 5 to 10 test cases that run your product's own code with a real model, and tries each check on made-up right and wrong replies. Makes the eval workflow run on pull_request and on push to the default branch (or only by hand, when they run on your machine), write a results file, and upload it with actions/upload-artifact as an artifact named evalship-results. Adds .evalship.yml, commits on a branch named evalship/setup and opens a pull request. When the evals need a key the repository doesn't have: whether they run in GitHub Actions, with the key added as a secret before the pull request runs, or on your machine
3. Review the evals Read only. Writes every flaw specific to the repository that holds up, most valuable first, each citing files and lines and ending in one concrete action, and up to 4 more upgrade steps. See The review. No
4. Send the review Pushes the review as the only file of a branch named evalship/review in your repository, where evalship reads it, then deletes the branch if it may. The review stays on GitHub until evalship reads it. No

After sending, the agent downloads where setup stands and sends you one plan: the pull requests it will open (one for the flaw fixes and upgrades, one for evals failing on the default branch), what goes in each, any product fixes listed apart, and how many merges that is for you. You go ahead, pick parts, or stop there and pick it up later. Then it:

  1. Opens each pull request, waits for its evals and evalship's comment, and acts on the comment's notes, pushing its changes in batches so each push's CI run counts.
  2. Sends you each one to merge, and goes on once you have.
  3. Tells evalship what it dismissed and did, in a file evalship-update.json on evalship/review.
  4. At the end, asks once to make the evalship check required and delete setup's merged branches. GitHub allows a required check on a private repository only with GitHub Pro (Team for an organization), so on its free plan the agent skips that and says so.

Besides that, it asks again only for a product fix the plan didn't name, or a decision only you can make.

It ends in at most three lines, with the link to your report. To pick setup up later, paste the same line again: once the repository has a review, it continues from where setup stands. Review shows that line too.

The agent never reads, prints, creates or changes secrets, never weakens, skips or deletes existing evals or assertions, and changes product code only to fix a bug an eval shows, once you've agreed to it in the plan. Every claim it makes about the repository cites a file and line it read.

4. Open your review

The agent links to your dashboard, https://evalship.com/gh/OWNER/REPO, once it sends the review. If you started from the setup page on evalship, that page opens it as soon as the review arrives.

Review follows setup's three steps and opens at the one setup is at:

Step Done when
Install GitHub App Done before the repo has a page on evalship.
Review evals The agent sent the review. It lists the flaws it found, most important first, each with the evidence and the fix.
Fix flaws Its rows are done, in order: results from the default branch reach evalship (once the setup PR merges), PR comments are on, every flaw is fixed by a merged pull request or dismissed (flaws evalship finds in setup's own pull requests count too), and every eval failing on the default branch is fixed or left for later. The page leads with the one thing to do next.

Then evalship comments on every pull request that changes your prompts, AI calls or evals, and sends a weekly report. The repo's page has five parts, each with a count of what needs you there. All issues lists what's broken now: eval runs that need you, and production's issues. Test has the eval runs, each with what to do next, and the week's insights, also at /gh/OWNER/REPO/test during setup. Monitor shows your app's AI calls once you connect production. Optimize has the changes your evals prove better, and your eval reviews. Settings has where the evals run, PR comments, the check, the README badge and the public dashboard. When a newer model from the same maker costs less per input and output token than one your evals run, by OpenRouter's prices, this week's insights suggest it, and Optimize lists it for your coding agent to try.

5. Add secrets if CI needs them

If your evals call a paid API in CI, the agent lists the secrets they need with a command for each, for example:

gh secret set OPENAI_API_KEY

If you chose to run them on your machine, there's nothing to add: your coding agent runs them there, for the setup pull request, for the default branch once it merges, and for each pull request when you ask. See Run evals on your machine.

6. Merge the setup PR

Merging runs the eval workflow on the default branch, and those results become the baseline. On your next pull request, an evalship check shows as in progress from the push on. Once the eval workflow finishes, the check has the results, and a comment with its notes follows. Until the notes are in, an evalship notes check shows as in progress, so you know to wait before you merge.

The setup pull request's own results have nothing to compare with, so its comment is a first pass: how many evals pass, with the ones that fail listed once as where your evals start, not as regressions. Its notes start with your next pull request, since your agent's review already covered the evals it adds. See How the PR comment works for baselines and what the comment contains.

Without a coding agent

You can set up CI by hand. Install the App at evalship.com/install or through the guided setup, then follow Connecting CI. You get PR comments, the check and badges, but no review of your evals until you run the setup prompt.

Next

  • Integrations: the steps for pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust and LangSmith.
  • Connecting CI: workflow requirements and examples per framework.
  • Result formats: JUnit XML, promptfoo JSON, pytest-json-report JSON, Vitest or Jest JSON, Inspect AI logs and evalship JSON.
  • Configuration: .evalship.yml, including how to make the check block merges.