Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

Articles · November 20, 2026

A Braintrust alternative that keeps evals in your repo

Braintrust is a hosted platform for evaluation and observability: tracing, experiments, datasets, a playground, prompt management, online scoring and Loop, its built-in agent. evalship does a smaller job. Your evals stay test files in your repository, evalship says which of them a pull request broke, and it checks every AI call in production. It's an alternative when that is what you need, and it runs next to Braintrust when you need the rest. This comparison is from both tools' docs and pricing pages as of November 20, 2026.

Braintrust and evalship side by side

Braintrust evalship
Where evals live Eval() calls in Braintrust's SDK, with datasets and experiments stored in Braintrust Test files in your repository, run by your own command: pytest, Vitest, Jest, promptfoo or tinyeval. evalship reads the results file
On a pull request Its GitHub Action keeps one comment: each score's average, its change against a base experiment, and how many test cases improved or regressed. A regression alone doesn't fail the job Compares each eval with its result on your default branch, sets aside flaky evals and failures within chance, and comments with what's left. Its check can block the merge
Finding gaps in your evals Loop writes evaluation criteria and builds datasets from production traces Your coding agent reviews every suite at setup and lists its flaws, such as a test case that can't fail or a rule no test case checks
In production Tracing and online scoring with the scorers you set. Topics clusters your logs to show patterns Checks every AI call with no scorer to write: failed calls, overpromises, made-up facts, private data, frustrated users. Each issue comes by email, with a line for your coding agent to fix it
Improving a prompt In Braintrust's playground, or with Loop Your coding agent tries prompt and model changes on your machine, and evalship says which win beyond chance
Playground, prompt management, human review, custom charts Yes, custom charts from Pro No. Prompts stay in your code, where a pull request shows every change
Self-hosting On Enterprise Not yet
Compliance SOC 2 Type II. SAML single sign-on on Enterprise Working towards SOC 2. Sign-in with GitHub or email
Price Starter: free, 1 GB of data, 10k scores, 14-day retention. Pro: $249 a month, 5 GB, 50k scores, 30-day retention. Both charge for use past that Free: 2 users, 1 private repository, 10,000 production calls a month, 30 days of eval history. Pro: $29 a month, unlimited users and repositories, 500,000 production calls, 1 year. Business: $99 a month

Braintrust has no limit on users on any plan. evalship is free for public repositories on every plan.

What differs on a pull request

Braintrust's eval-action comes closer to a comparison with main than most tools: it shows each score's change against an earlier experiment. Two things are left to you.

  1. The job stays green on a regression. The action fails only when an eval throws. A gate on scores is a custom reporter you write.
  2. The counts include chance. In the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?). "4 regressions" can be four test cases that move on every run, or four that never moved before. One earlier run can't tell them apart.

evalship compares each eval with its own history on your default branch. It sets aside the ones that are flaky on main and the failures within the noise it measures over each suite's last 30 runs, and with the gate on, its check fails a pull request that breaks an eval that passed on main (how the PR comment works).

When evalship fits better

  • Your evals belong with your code. They are test files, reviewed in the same pull request as the prompt they test, with no SDK between your test runner and your CI.
  • You want a check that blocks a real break and ignores a flaky one.
  • You don't know whether your evals would catch a break. In 10 of 10 open-source eval suites, at least one test case couldn't fail (the flaws we found).
  • You want every plan to have everything. Pull request checks, production checks and experiments are on evalship's Free plan too. The plans differ in users, private repositories, volume and how long history is kept (pricing).

When to stay with Braintrust

  • People iterate on prompts in a playground, product managers and domain experts included. evalship has no playground: a prompt changes in a pull request.
  • Your team reviews and labels outputs by hand. Braintrust has human review scores. evalship has no human review step.
  • You want Loop or Topics: an agent inside the platform that builds datasets from traces, and clustering across all your logs.
  • Your security review asks for SOC 2 Type II, SAML single sign-on or a HIPAA agreement. Braintrust has them, the last two on Enterprise.
  • You want your own charts over traces and scores.

Using both

Nothing has to move. Your eval script keeps reporting to Braintrust and also writes a results file, about 15 lines more. evalship then compares every pull request with main, and each changed eval links to its trace in Braintrust (Braintrust evals on every pull request).

To try evalship, paste one line into your coding agent: Set up https://evalship.com/setup.md (quickstart).

Further reading