LLM evals with Vitest in GitHub Actions

TypeScript teams often write their LLM evals as Vitest tests: call the model, score the answer, expect the score to clear a threshold. Vitest makes that easy to write. In CI, it reports each eval as passed or failed, and on a pull request that isn't the question. The question is whether this change broke an eval that was passing on main.

A red run doesn't say what broke, and a retry hides it

vitest run exits with code 1 when any test fails. LLM evals fail by chance: in the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?). At that rate, a suite of 50 evals goes red on 72% of pull requests that change nothing (the table).

The usual fix is retry. It turns the run green, and it hides the flake: in the JUnit report, a test that failed once and passed on its second try is a plain pass, with nothing to say it was retried. An eval that passes 1 time in 2 then looks as solid as one that never fails, and a change that makes it worse looks like nothing.

What tells chance from a real break is each eval's own history on your default branch. Without retries, that history keeps every failure, so it can tell an eval that flips from one that broke.

Write JUnit XML and keep the console

name: evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      - run: npm ci
      - run: npx vitest run evals --reporter=default --reporter=junit --outputFile.junit=eval-results.xml
        continue-on-error: true # the comparison with main decides, not one failure
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-results
          path: eval-results.xml

--reporter=default next to --reporter=junit keeps the usual console output. With --reporter=junit alone, the log only says where the report went.
Naming any reporter turns off Vitest's GitHub Actions annotations, the failures it marks in the pull request's files. Add --reporter=github-actions to keep them.
push to main gives every pull request results to compare with, and if: always() uploads the file when evals fail.

Record the score with annotate

A pass keeps no score. A score that drops from 0.95 to 0.72 against a threshold of 0.7 still passes, and that's the drift worth seeing before it fails. Since Vitest 3.2, a test can annotate itself, and the JUnit reporter writes each annotation as a property of the test: its type becomes the name, its message the value.

import { test, expect } from "vitest"

test("declines a refund above the limit", async ({ annotate }) => {
  const reply = await answer("Can I get a refund of $500?")
  const score = await judgeRefundPolicy(reply)
  await annotate(String(score), "score")
  await annotate("0.7", "threshold")
  expect(score).toBeGreaterThanOrEqual(0.7)
})

The XML then carries <property name="score" value="0.41"/> on the test. Annotate before the expect: a failed expect ends the test, and annotating from afterEach or onTestFinished throws and fails the test.

Give each eval a name that lasts

A JUnit result is known by its file path and its test name, describe blocks included. Rename either and the eval starts over with no history. Keep changing values out of names: no dates, no model names, no inputs pasted into a template string. For a table of cases, name each one, and use test.for rather than test.each, since only test.for hands the test its annotate:

const cases = [
  { id: "refund-above-limit", question: "Can I get a refund of $500?", expected: "decline" },
  { id: "refund-below-limit", question: "Can I get a refund of $50?", expected: "approve" },
]

test.for(cases)("$id", async ({ question, expected }, { annotate }) => { /* ... */ })

Make it faster

An eval spends most of its time waiting on the model. test.concurrent runs a file's tests together, but then a test's printed output can land under another test in the JUnit report. For a long suite, split the cases across jobs with a strategy.matrix and give each job's upload its own name. Slow evals has the details.

vitest-evals and evalite

vitest-evals runs its evals as ordinary Vitest tests, so each case is a test case in the JUnit report. Its scores stay in Vitest's task metadata rather than in the XML, so annotate the ones you want to keep. With judges set and no judgeThreshold, a case needs a perfect score to pass.
evalite is built on Vitest but runs its own command and reporter, and writes its results as JSON (--outputPath), not JUnit XML. Run under plain vitest, its evals fail.

How evalship does this

evalship reads the eval-results.xml your workflow uploads and compares every pull request with your default branch, eval by eval. It sets aside evals that couldn't run (a missing key, a rate limit, a retired model), evals that are flaky on main, and failures within the noise it measures over each suite's last 30 runs. It reads the score and threshold you annotate, so a drop shows before the expect fails. Then it comments on the pull request with what's left. How the PR comment works has the details.

It also flags the prompts and LLM calls a pull request changes that no eval covers. Annotate targets (await annotate("src/support-bot.ts#answer", "targets")) to say which code an eval exercises; see Untested changes.

With this in .evalship.yml, its check fails a pull request that breaks an eval that passed on main:

gate:
  fail_on_new_failures: true

evalship is free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart). The agent writes the workflow above for your repository.