LLM evals with pytest in GitHub Actions

Most Python teams write their first LLM evals as pytest tests: call the model, score the answer, assert that the score clears a threshold. On a pull request, that assert has to answer a question it wasn't built for: did this change break the eval, or would it have failed anyway?

pytest's exit code is wrong both ways

pytest exits with code 1 when any test fails. For unit tests that's right. For LLM evals, it goes red when nothing broke and stays green when something did.

Red when nothing broke. LLM evals fail by chance. In the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?). At that rate, a suite of 50 evals goes red on 72% of pull requests that change nothing (the table). Failures that never reached the model count the same: on PrefectHQ/prefect-mcp-server#200, 14 of the 20 pytest evals failed with the same 404, because the judge model had been retired (case study).

Green when something broke. The usual way out of the red is an eval step that reports without failing the build. HolmesGPT's eval workflow works that way. When a pull request commented out part of its system prompt, one eval that had passed every run on master broke, and the check stayed green (case study).

What tells the two apart is each eval's own history on your default branch. The rest of this guide sets up the results that comparison needs.

Run the evals on main too, and keep the results

name: evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/evals --junitxml=eval-results.xml -o junit_family=xunit1
        continue-on-error: true # the comparison with main decides, not one failure
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-results
          path: eval-results.xml

push to main gives every pull request results to compare with. Without it, each pull request looks like a first run.
--junitxml writes one <testcase> per eval, with its failure message and duration.
if: always() uploads the file when evals fail, the run you most want to read.

Record the score, not only the assert

An assert keeps pass or fail. A score that drops from 0.95 to 0.72 against a threshold of 0.7 still passes, and that's the drift worth seeing before it fails. record_property writes the score into the XML:

def test_refund_limit(record_property):
    reply = answer("Can I get a refund of $500?")
    score = judge_refund_policy(reply)
    record_property("score", score)
    record_property("threshold", 0.7)
    assert score >= 0.7

Since pytest 6.0, the XML follows the xunit2 format by default. It keeps the properties, but pytest warns on every test that records one (record_property is incompatible with junit_family 'xunit2'), and in a project that turns warnings into errors, those tests error before they run. xunit2 also leaves out each test's file path. Switch to xunit1 once, in pyproject.toml:

[tool.pytest.ini_options]
junit_family = "xunit1"

Give each eval a name that lasts

A JUnit result is known by its module path and test name. For a parametrized test, pytest puts the parameter in the name, as in test_refund[Can I get a refund of $500?]. Edit the question and it becomes a different test with no history. Give each case an id:

CASES = [
    pytest.param("Can I get a refund of $500?", "decline", id="refund-above-limit"),
    pytest.param("Can I get a refund of $50?", "approve", id="refund-below-limit"),
]

@pytest.mark.parametrize("question, expected", CASES)
def test_refund(question, expected, record_property): ...

Moving or renaming the test file changes the name of every eval in it, too.

Keep errors apart from wrong answers

A rate limit, a missing key and a retired model all raise inside the test, and pytest reports each as a failure like a wrong answer. Two of them you can see coming:

Pull requests from forks and Dependabot get no secrets, so every eval fails with an authentication error. Run the eval job on those only when it can reach the model: if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository.
A judge you didn't choose. When a library grades with a model, name the model in your code, so the judge doesn't change when the library's default does. Scores from two judges measure different things.

DeepEval

DeepEval test files are pytest files, and they run under plain pytest with the same flags. deepeval test run passes its extra arguments to pytest too, but its -o is its own flag, so set junit_family in pyproject.toml rather than with -o. A metric's score reaches the XML only when you record it, so measure the metric yourself instead of calling assert_test:

def test_refund_answer(record_property):
    question = "Can I get a refund of $500?"
    case = LLMTestCase(input=question, actual_output=answer(question))
    metric = AnswerRelevancyMetric(threshold=0.7, model="gpt-4.1")
    metric.measure(case)
    record_property("score", metric.score)
    record_property("threshold", metric.threshold)
    assert metric.is_successful(), metric.reason

Make it faster

An eval spends most of its time waiting on the model, and pytest runs one test at a time. pytest-xdist runs several: pytest -n 4 takes about a quarter of the time, as far as your provider's rate limit allows, and still writes one XML file.

How evalship does this

evalship reads the eval-results.xml your workflow uploads and compares every pull request with your default branch, eval by eval. It sets aside evals that couldn't run (a missing key, a rate limit, a retired model), evals that are flaky on main, and failures within the noise it measures over each suite's last 30 runs. It reads score and threshold from record_property, so a drop shows before the assert fails. Then it comments on the pull request with what's left. How the PR comment works has the details.

It also flags the prompts and LLM calls a pull request changes that no eval covers. Record targets (record_property("targets", "app/support_bot.py#answer")) to say which code an eval exercises; see Untested changes.

With this in .evalship.yml, its check fails a pull request that breaks an eval that passed on main:

gate:
  fail_on_new_failures: true

evalship is free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart). The agent writes the workflow above for your repository.