LLM evals on pull requests: how the tools compare (2026)
A change to a prompt, a model or an agent can break answers no unit test sees. Evals on every pull request catch that, if the check answers one question: did this change break something that worked on main? Here is how the main eval tools answer it, from their docs, GitHub Actions and source code as of November 2, 2026.
What a pull request check needs
- A run on main to compare with. A failing test case means little until you know it passed before.
- A result per test case. An average can hold steady while one case breaks.
- A way to tell chance from a break. In the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?).
- A rule that fails the check, so a break can block the merge.
The tools side by side
| Tool | Runs in CI with | On the pull request | Compares with main | Fails the check when |
|---|---|---|---|---|
| promptfoo | promptfoo/promptfoo-action, or the CLI |
A new comment each run, with pass and fail counts | No, it runs the pull request's code only | Any test fails, unless you set a pass-rate threshold |
| Braintrust | braintrustdata/eval-action |
One comment, updated: each score's average and change, and how many cases improved or regressed | Yes, against the newest experiment on main's nearest ancestor commit | An eval throws. A regression alone doesn't fail it |
| Langfuse | langfuse/experiment-action |
One comment, updated: each script's status, its scores, and up to 50 items | No, your script sets thresholds or checks a baseline file you review | Your script raises RegressionError, or crashes |
| LangSmith | Its pytest, Vitest and Jest plugins. No official Action | Not in the docs. The example repository builds its own comment | Not in the docs. The example checks fixed criteria | A test assertion fails |
| DeepEval | deepeval test run, which wraps pytest |
Not in the docs | No, each metric has a fixed threshold | A metric is below its threshold |
| Confident AI PR Eval Gate | An Action plus a GitHub App | A check run, not a comment | Yes, against the base branch | A metric's average drops more than a set tolerance |
| Raindrop Simulations | Early access, no docs yet | Product-page mockups show scenarios, regressions and cost | It replays production traces against the change | Mockups show "merge blocked" |
| pytest, Vitest or Jest alone | Your test command, writing JUnit XML | Nothing, unless you add a reporter | No | A test fails |
OpenAI Evals isn't in the table. Its dashboard and API shut down on November 30, 2026, and OpenAI recommends promptfoo instead (Replacing OpenAI Evals).
What each one needs
| Tool | Account needed in CI | Free tier |
|---|---|---|
| promptfoo | No | Open source. The Community plan has no eval cap |
| Braintrust | Yes, an API key | Starter: 1 GB of data, 10k scores, 14-day retention |
| Langfuse | Yes, on Langfuse Cloud or your own instance | Self-hosting is free. Cloud Hobby: 50k units a month |
| LangSmith | No, with LANGSMITH_TEST_TRACKING=false |
Developer: 1 seat, 5k traces a month |
| DeepEval and Confident AI | Not for DeepEval. Yes for the PR Eval Gate | DeepEval is open source. Confident Free: 5 test runs a week |
| Raindrop | Yes | Hobby: 1,000 events a month. Simulations has no listed price |
promptfoo
The Action runs promptfoo eval once, on the pull request's code. Its README says it "does not run separate base and head evaluations". Any failed test fails the job, because the pass-rate threshold defaults to 100%. repeat and repeat-min-pass run each test several times and pass it when enough of the runs pass. OpenAI announced in March 2026 that it is acquiring promptfoo, which stays open source.
Sources: promptfoo-action, command line docs. Our guide: Replacing OpenAI Evals: promptfoo in CI without flaky reds.
Braintrust
Braintrust comes closest to a comparison with main. The comment shows each score's change against a base experiment: by default the newest one on the nearest ancestor commit of your default branch, so main needs runs of its own. It counts the test cases that improved and regressed. A regression doesn't fail the Action, only an eval that throws does. A custom reporter can gate on scores. trialCount runs each input several times.
Sources: eval-action, Run evals in CI. Our guide: Braintrust evals on every pull request.
Langfuse
The experiment action runs your experiment scripts and keeps one comment: each script's status, its scores, and a table of up to 50 items. There's no automatic baseline. Your script decides, and raises RegressionError to fail the job. Langfuse's docs recommend a reviewed baseline file in the repository, and warn that an aggregate threshold can miss a newly failing case.
Sources: experiment-action, Experiments in CI/CD. Our guide: Langfuse experiments on every pull request.
LangSmith
LangSmith runs inside pytest, Vitest or Jest, so a failed assertion fails the job. It has no official GitHub Action; one was announced in January 2025. Its example repository, langchain-ai/cicd-pipeline-example, posts its own comment that checks each score against fixed criteria.
Sources: pytest, Vitest and Jest. Our guide: LangSmith evaluations on every pull request.
DeepEval and Confident AI
deepeval test run wraps pytest, and a metric below its threshold fails the test. -r repeats each test case, and flaky=True turns a failure into a warning. Confident AI's PR Eval Gate adds a comparison: a check run that fails when a metric's average drops more than a set tolerance against the base branch. The Gate calls each test case once.
Sources: DeepEval in CI/CD, PR Eval Gate. Our guide: LLM evals with pytest.
Raindrop Simulations
Raindrop replays the production traces most relevant to a pull request against its change, in simulated services. It's in early access and has no docs page yet, so what it shows on a pull request comes from mockups on its product page.
Source: raindrop.ai/simulate.
What none of them does
Of the tools with docs, none compares each test case with its own history on main. Braintrust compares with one earlier run, and Confident AI with an average. promptfoo and DeepEval repeat a test case within the pull request's run. Repeats smooth out one noisy sample. They don't tell a case that broke from one that fails on main every few runs, and an average hides a single case that broke.
Where evalship fits
evalship doesn't run your evals. It reads the results your tool already writes, from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust or LangSmith, and compares every pull request with your default branch, test case by test case. It sets aside results that couldn't run, cases that are flaky on main, and failures within the noise it measures over each suite's last 30 runs. Then it comments with what's left, and its check can fail a pull request that breaks a case that passed on main (how the PR comment works).
evalship is free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart).