Quickstart
evalship comments on every pull request with what your LLM eval results mean. It does not run your evals: your GitHub Actions workflow runs them and uploads a results file, and evalship reads that file when the workflow finishes.
Setup is a one-line prompt you paste into a coding agent inside your repository. The agent connects CI in a pull request and sends evalship a review of your evals. It asks before every change.
Requirements
- A GitHub repository with evals, or with LLM calls you want evals for.
- GitHub Actions. Other CI systems are not supported yet.
- A coding agent that can read the repository and run shell commands: Claude Code, Cursor, Codex, Copilot or Windsurf. The agent uses
git,ghandcurl.
Public repositories are free. Private repositories are free during the beta; see Pricing and billing.
1. Choose where to start
| Start from | How it goes |
|---|---|
| The homepage | Copy the one-liner on evalship.com. Your agent ends with a claim link that connects the repository. |
| The guided setup | Click "Get started" or "sign in with GitHub" to open evalship.com/start: 1 Connect GitHub, 2 Pick a repo, 3 Set up CI, 4 Done. No claim link. |
Both install the evalship GitHub App, which GitHub lists as evalship-app (GitHub reserves the name "evalship"). The check it adds to commits is called evalship. The permissions it asks for are listed in Data and privacy.
2. Paste the one-liner into your agent
Start your agent in the repository and paste:
Set up evalship: run `curl -sL https://evalship.com/setup.md` and follow the steps.
In the guided setup, step 3 gives you this line for the repository you picked. Its URL carries a setup key, so the review attaches to that repository directly.
The agent downloads the full instructions with curl, because a web fetch could summarize them and drop steps. You can read them at evalship.com/setup.md (/prompt redirects there). The first line shows the prompt version, and the review records it.
3. Work through the steps with the agent
The agent tells you what it is about to do, then works through four steps.
| Step | What the agent does | Asks you |
|---|---|---|
| 1. Find the evals | Read only. Looks for promptfoo, Evalite, vitest-evals, DeepEval, Braintrust, LangSmith, Inspect AI, OpenAI Evals, Ragas, Opik, Langfuse experiments and hand-rolled evals, the prompts and LLM calls they cover, and the workflows that run them. | No |
| 2. Connect CI | Makes the eval workflow run on pull_request and on push to the default branch, write a results file, and upload it with actions/upload-artifact as an artifact named evalship-results. Commits on a branch named evalship/setup and opens a pull request. Lists the secrets CI needs. |
Before creating .github/workflows/evals.yml if no workflow runs the evals, and before committing |
| 3. Review the evals | Read only. Writes at most 8 findings specific to the repository, each citing files and lines and ending in one concrete action, and an upgrade plan of 3 to 6 steps. See The audit. | No |
| 4. Send the review | Writes the review as JSON, shows it to you, and POSTs it to https://evalship.com/api/v1/audits. |
Before sending |
After that, the agent offers two extras and does only what you pick: apply the proposed upgrade steps, and add the README badge if the repository is public and its README already has a row of badges.
The agent never reads, prints, creates or changes secrets, never weakens, skips or deletes existing evals or assertions, and never changes product code. Every claim it makes about the repository cites a file and line it read.
4. Open the setup page
If you started from the homepage, the agent ends with a link like https://evalship.com/claim/.... Opening it installs the GitHub App on the repository (for a new user, one GitHub screen that installs and authorizes it) and attaches the review. The link works for 7 days, and only for someone with access to the repository on GitHub.
Both paths end on the setup page for the repository, with a live checklist:
| Item | Done when |
|---|---|
| Review received | The agent sent the review. |
| Setup PR open | The review names the setup PR, or evalship has results for the repository. |
| First results from GitHub Actions | The eval workflow ran on the setup PR or the default branch. |
When the first results arrive, the page moves to Done.
5. Add secrets if CI needs them
If your evals call a paid API in CI, the agent lists the secrets they need with a command for each, for example:
gh secret set OPENAI_API_KEY
The agent never sets secrets itself.
6. Merge the setup PR
Merging runs the eval workflow on the default branch, and those results become the baseline. On your next pull request, once the eval workflow finishes, evalship posts a comment and an evalship check on the commit.
The first results evalship sees for a repository have nothing to compare against, so that comment only lists them. See How PR analysis works for baselines and what the comment contains.
Without a coding agent
You can set up CI by hand. Install the App at evalship.com/install or through the guided setup, then follow Connecting CI. You get PR comments, the check and badges, but no review of your evals until you run the setup prompt.
Next
- Connecting CI: workflow requirements and examples per framework.
- Result formats: JUnit XML, promptfoo JSON and evalship JSON.
- Configuration:
.evalship.yml, including how to make the check block merges.