Skip to content

FAQ

Does evalship run my evals?

No. Your GitHub Actions workflow runs them, with your API keys, and uploads a results file as an artifact. evalship reads the artifact when the workflow finishes. See Connecting CI. To keep your keys off GitHub, your coding agent runs them on your machine and sends the results instead: Run evals on your machine.

Can my coding agent check a change before it opens a pull request?

Yes. Setup adds a line to your agent's instructions: before it finishes a change to a prompt, model or AI call, it runs the evals the change affects on your machine, committed or not, and sends evalship the diff with the results. evalship compares them with your default branch and writes the same notes a pull request gets. See Before you finish.

Does evalship change my prompts or models?

Only what you pick, through your coding agent. Optimize has your agent try changes on your machine, such as a cheaper model or a clearer rule, and evalship says which win beyond chance, and by how much. A winner becomes a pull request you merge, or don't.

Can my Claude plan pay for the evals?

If your evals call Claude, yes: Claude Max, Ultra and Team plans include $100 to $500 a month of API credit for a key you use in CI. See Run your Claude evals in CI on your Claude plan.

Do I need an eval framework?

No. A test file that calls your model and asserts on the reply is enough, and so is a script that scores replies (the setup agent makes it write a results file). If your test runner writes JUnit XML (pytest, Vitest, Jest, RSpec, Go, Java, PHPUnit and Rust tests all can), evalship reads it as is. The setup review looks at whatever you have and says what to make stricter first.

I only have a few evals, or none. Is this for me?

Yes. A handful of evals is where one that can't fail hurts most, and the review finds those. With no evals, setup offers to write a first small suite for your most important AI call, the review names what to test next, and every pull request still shows the AI calls it adds or changes that nothing tests.

Which eval frameworks work?

Any that can produce JUnit XML, promptfoo JSON or evalship JSON. pytest (including DeepEval tests run with pytest), Vitest, Jest, RSpec, Go, Java (Maven or Gradle), PHPUnit and Rust (cargo-nextest) tests write JUnit XML; promptfoo writes its own JSON. A bash script can write evalship JSON with jq (example). For frameworks without either, such as Braintrust, LangSmith, Langfuse experiments or Inspect AI, write evalship JSON from the results in your eval script (examples). The setup agent does this for you when it's needed.

Does it work with CI other than GitHub Actions?

Not yet. evalship reads results from GitHub Actions workflow artifacts in the same repository, or sent from your machine.

Why does setup need my coding agent?

Setup means reading the repository, understanding which evals exist and what they cover, and changing workflows and harness code. A coding agent in your repository already has that context and can open the setup PR. It also means the review is written on your machine with your own model subscription, which is why it's free.

You can skip the agent: install the App at evalship.com/install and set up CI by hand. You get PR comments without the review.

What will my agent do in my repository?

It asks before it starts, sends one plan before it changes anything, and never merges, force-pushes or reads your secrets. What your agent does lists every step, with the full prompt.

What if my agent makes things up in the review?

evalship checks every file, line and quote each flaw cites against your default branch, and hides flaws whose citations don't hold. Review says how many were hidden. A repository member can dismiss a wrong flaw with a reason, and reopen it. See The audit.

Is my code sent anywhere?

evalship reads files through the GitHub API to find AI calls in pull requests and to check the review's citations. It doesn't store your source code, only pull request descriptions and excerpts of the diffs to eval, prompt and workflow files. When a pull request changes prompts, AI calls or evals, or has new failures its rules can't explain, it sends the diffs that matter for the evals, up to 4,000 characters per file, your review and excerpts of the failing evals to a language model (today Claude Haiku 5.5, through Anthropic's API) for notes on the change. Owners can turn model notes off in the account's settings, and then nothing goes to a model. Your coding agent sends your code to its own model provider under that provider's terms. Details are in Security and Data and privacy.

Why is the evalship check neutral?

Because .evalship.yml sets no gate, and a neutral check never blocks a merge. Setup adds gate.fail_on_new_failures: true to the setup PR; if you skipped that, set gate.fail_on_new_failures, gate.max_pass_rate_drop or a required suite in .evalship.yml, and require the evalship check in your branch protection rules. On a private repository, GitHub offers those rules only with GitHub Pro (Team for an organization): without it, the check still reports but can't block a merge.

I have a monorepo, or several eval workflows. Does that work?

Yes. evalship groups results by commit: artifacts from every workflow run for the same commit merge into one run, and the comment is updated as each workflow finishes. Keep suite names distinct across workflows, because two results with the same suite and key overwrite each other. Use results.artifacts to read only some artifacts, comparison.suites.<name>.required to require specific suites, and ?suite= for per-suite badges.

We already use Langfuse, Braintrust or LangSmith. Do we need to switch?

No, keep them. evalship adds what they leave out, and it lives in GitHub instead of in another dashboard:

  • The review checks your evals themselves. A platform shows a score of 0.92. It doesn't say that the judge counts errors as passes, or that no test case checks the refund limit in your system prompt.
  • Pull requests show the prompts and AI calls that nothing tests. Platforms see traces, not your diff.
  • Real regressions are separated from flaky evals using each eval's history on your default branch.
  • One comment covers every setup. If promptfoo, pytest and Langfuse experiments each run some of your evals, their results land in the same comment and check.

evalship doesn't run evals, store datasets, trace production traffic or manage prompts, and it isn't going to. A prompt you edit in your platform's prompt management never shows up in a diff, so evalship can't flag it as untested. Your eval script keeps reporting to your platform and also writes a results file in CI. Give each eval a url and every changed eval in the PR comment links to its trace. Integrations has the code.

Is evalship open source?

Not yet. What runs in your repository you can read: What your agent does has the full setup prompt.

Can I self-host evalship?

Not yet. If your team needs to run evalship on your own infrastructure, for example because diffs can't leave your network, email hello@evalship.com with what you'd need. Those requests decide when we offer it.

How do I stop the PR comments?

Turn off PR comments on the repository's Settings page, or set comment.enabled: false in .evalship.yml: either stops the comments and keeps the check. Set comment.only_on_change: true to skip comments when nothing changed. To stop everything, remove the repository from the evalship App's repository access in your GitHub settings, or uninstall the App, which also cancels a paid plan at the end of its billing period.