Skip to content

Articles · October 30, 2026

Braintrust evals on every pull request

Braintrust keeps your datasets, your experiments and the score changes between them. Running its evals on every pull request adds a question its comparison answers only in part: which test cases did this change break, and which moved by chance?

What Braintrust's eval action reports

Braintrust's guide to running evals in CI uses braintrustdata/eval-action. It runs your evals and keeps one comment on the pull request. For each experiment, the comment lists every score with its average, its change against a baseline experiment (as in "83% (+3pp)"), and how many test cases improved and regressed. The baseline is an experiment you set in Braintrust, or one it picks from your git history.

The action fails the job only when the eval command fails, which happens when an eval throws. It has no score thresholds of its own. A gate on scores is a custom Reporter() you write.

The counts include chance. LLM evals fail by chance: in the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?). "4 regressions" on a pull request can be 4 test cases that move on every run, or 4 that never moved before. The comment can't say which, because that takes each test case's history over many runs of your default branch.

HolmesGPT shows the result. Its pytest evals log to Braintrust, and its bot comments on pull requests with each test case's status and changes against master in time, cost, tokens and tool calls. The eval step has continue-on-error: true, so the build never fails on an eval. When a pull request commented out part of its system prompt, one eval that had passed every run on master broke, and the check stayed green (case study).

Run the eval from a script and write the results

Run the eval with python on pull requests and on pushes to your default branch. Under braintrust eval, and so under the eval action, Eval() only registers the evaluator and returns no results, so there's nothing to write. Run with python, it still logs an experiment to Braintrust, so its comparison view keeps working.

import json
from braintrust import Eval, init_dataset
from app.support_bot import answer

THRESHOLD = 0.8

def task(input, hooks):
    hooks.metadata["trace_url"] = hooks.span.link()  # this test case's trace in Braintrust
    return answer(input)

def exact(input, output, expected):
    return 1.0 if output == expected else 0.0

result = Eval("support-bot", data=init_dataset("support-bot", "golden"), task=task, scores=[exact])

evals = []
for r in result.results:
    scores = [s for s in r.scores.values() if s is not None]
    score = min(scores) if scores else None
    evals.append({"key": (r.origin or {}).get("id") or r.metadata["id"], "score": score, "threshold": THRESHOLD,
                  "passed": r.error is None and score is not None and score >= THRESHOLD,
                  "failure": str(r.error) if r.error else None,
                  "input": r.input, "output": r.output, "url": r.metadata.get("trace_url")})

json.dump({"schema": "evalship/v1", "suite": "support-bot", "url": result.summary.experiment_url, "evals": evals},
          open("eval-results.json", "w"), default=str)

name: evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python -m evals.support_bot_eval # -m, so the script can import app/ from the repository root
        env:
          BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-results
          path: eval-results.json

What the script takes care of

  • Test cases whose task raised. They come back with no scores and with r.error set, so they count as failed, with the error as the reason.
  • The lowest score, not the average. With several scorers, one that fails fails the test case. An average can hide a hard rule that broke. Keep scores between 0 and 1.
  • Keys that last. Test cases from a Braintrust dataset carry their record id in r.origin, which stays the same while you edit the dataset. With data of your own, give each record "metadata": {"id": "..."}. A key that changes starts the case's history over.
  • TypeScript. The same fields exist in camelCase, and the experiment's link is result.summary.experimentUrl.

How evalship does this

evalship reads the eval-results.json your workflow uploads and compares every pull request with your default branch, test case by test case. It sets aside cases that couldn't run (a missing key, a rate limit, a retired model), cases that are flaky on main, and failures within the noise it measures over each suite's last 30 runs. Then it comments on the pull request with what's left, and each changed case links to its trace in Braintrust. How the PR comment works has the details.

With this in .evalship.yml, its check fails a pull request that breaks a case that passed on main:

gate:
  fail_on_new_failures: true

evalship never sees your Braintrust key: the script runs in your own GitHub Actions, and evalship reads only the file it uploads. It's free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart). The agent writes the script and the workflow for your repository.

Further reading