FAQ

Does evalship run my evals?

No. Your GitHub Actions workflow runs them, with your API keys, and uploads a results file as an artifact. evalship reads the artifact when the workflow finishes. See Connecting CI.

Which eval frameworks work?

Any that can produce JUnit XML, promptfoo JSON or evalship JSON. pytest (including DeepEval tests run with pytest), Vitest, Jest, RSpec and Go tests write JUnit XML; promptfoo writes its own JSON. For frameworks without either, such as Braintrust, LangSmith or Inspect AI, write evalship JSON from the results in your eval script. The setup agent does this for you when it's needed.

Does it work with CI other than GitHub Actions?

Not yet. evalship reads results only from GitHub Actions workflow artifacts in the same repository.

Why does setup need my coding agent?

Setup means reading the repository, understanding which evals exist and what they cover, and changing workflows and harness code. A coding agent in your repository already has that context and can open the setup PR. It also means evalship never runs a model over your code: the audit is written on your machine with your own model subscription, which is why it's free.

You can skip the agent: install the App at evalship.com/install and set up CI by hand. You get PR comments without the audit.

What if my agent makes things up in the audit?

evalship checks every file, line and quote each finding cites against your default branch, and hides findings whose citations don't hold. The report says how many were hidden. You can also vote on each finding. See The audit.

Is my code sent anywhere?

evalship reads files through the GitHub API to find LLM calls in pull requests and to check audit citations, and doesn't store source code. When a run has new failures, it sends excerpts of the failing cases and the diffs of changed prompt and LLM files, up to 1,500 characters per file, to Anthropic's API to name failure patterns. Your coding agent sends your code to its own model provider under that provider's terms. Details are in Data and privacy.

Why is the evalship check neutral?

Because it is advisory by default: a neutral check never blocks a merge. To make it pass or fail, set gate.fail_on_new_failures, gate.max_pass_rate_drop or a required suite in .evalship.yml, and require the evalship check in your branch protection rules.

I have a monorepo, or several eval workflows. Does that work?

Yes. evalship groups results by commit: artifacts from every workflow run for the same commit merge into one run, and the comment is updated as each workflow finishes. Keep suite names distinct across workflows, because two results with the same suite and key overwrite each other. Use results.artifacts to read only some artifacts, comparison.suites.<name>.required to require specific suites, and ?suite= for per-suite badges.

How is this different from Braintrust, LangSmith or the promptfoo dashboard?

Those tools run evals and keep experiments, datasets and traces in their own platforms, where you go to look at them. evalship doesn't run evals or store datasets. It reads the results your CI already produces and reviews them in the pull request: what regressed against the baseline, which failures are flaky noise, what the new failures have in common, and which changed prompts no eval exercises. You can use both.

Can I self-host evalship?

No. evalship is only available as the hosted service at evalship.com.

How do I stop the PR comments?

Set comment.enabled: false in .evalship.yml to stop comments and keep the check. Set comment.only_on_change: true to skip comments when nothing changed. To stop everything, remove the repository from the evalship App's repository access in your GitHub settings, or uninstall the App.