LLM evals with Jest in GitHub Actions

Plenty of LLM evals live in a Jest suite: call the model, score the answer, expect the score to clear a threshold. On a pull request, Jest says which tests failed. What you need to know is whether this change broke an eval that was passing on main, or whether it would have failed anyway.

A red run doesn't say what broke, and a retry hides it

Jest exits with code 1 when any test fails. LLM evals fail by chance: in the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?). At that rate, a suite of 50 evals goes red on 72% of pull requests that change nothing (the table).

jest.retryTimes(2) turns the run green, and it hides the flake. With jest-junit, a test that failed once and passed on its second try is a plain pass in the XML, with nothing to say it was retried. An eval that passes 1 time in 2 then looks as solid as one that never fails.

What tells chance from a real break is each eval's own history on your default branch, with every failure in it. The rest of this guide sets up the results that comparison needs.

Write JUnit XML with names that hold

Jest writes JUnit XML through jest-junit, installed as a dev dependency. Its defaults put the describe block and the test name together in both the class name and the test name, leave out the file, and name a suite undefined when its file has no describe block. Five settings fix that:

name: evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      - run: npm ci
      - run: npx jest evals --ci --reporters=default --reporters=jest-junit
        continue-on-error: true # the comparison with main decides, not one failure
        env:
          JEST_JUNIT_OUTPUT_FILE: eval-results.xml
          JEST_JUNIT_ADD_FILE_ATTRIBUTE: "true"
          JEST_JUNIT_SUITE_NAME: "{filepath}"
          JEST_JUNIT_CLASSNAME: "{classname}"
          JEST_JUNIT_TITLE: "{title}"
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-results
          path: eval-results.xml

--reporters=default keeps the usual console output next to the XML.
push to main gives every pull request results to compare with, and if: always() uploads the file when evals fail.
Names: each eval is then known by its describe blocks and test name, in a suite named after its file. Keep changing values out of test names, and name each case in a table: test.each(cases)("$id", ...). Renaming a file or a test starts its eval over with no history.

Record the score

A pass keeps no score. A score that drops from 0.95 to 0.72 against a threshold of 0.7 still passes, and that's the drift worth seeing before it fails. Jest has no way for a test to attach data to its own result, but jest-junit reads per-test properties from a function you write. A side file connects the two, in three small files:

// evals/record.js: a test calls record({ score, threshold }) before its expect
const fs = require("node:fs")
const FILE = "reports/eval-properties.jsonl"
exports.FILE = FILE
exports.record = (props) => {
  fs.mkdirSync("reports", { recursive: true })
  fs.appendFileSync(FILE, JSON.stringify({ test: expect.getState().currentTestName, ...props }) + "\n")
}

// junitTestCaseProperties.js, next to package.json: jest-junit calls it once per test
const fs = require("node:fs")
const { FILE } = require("./evals/record")
const byTest = new Map(
  (fs.existsSync(FILE) ? fs.readFileSync(FILE, "utf8").trim().split("\n") : [])
    .filter(Boolean).map((line) => JSON.parse(line)).map(({ test, ...props }) => [test, props]),
)
module.exports = (testResult) => byTest.get(testResult.fullName) ?? {}

// jest.global-setup.js, set as globalSetup in your Jest config: clears the last run's scores
module.exports = () => require("node:fs").rmSync(require("./evals/record").FILE, { force: true })

A test then records its score before the expect:

const { record } = require("./record")

test("declines a refund above the limit", async () => {
  const reply = await answer("Can I get a refund of $500?")
  const score = await judgeRefundPolicy(reply)
  record({ score, threshold: 0.7 })
  expect(score).toBeGreaterThanOrEqual(0.7)
})

Each test in the XML gets <property name="score" value="0.82"/>. This works across Jest's worker processes. It matches tests by their full name, so give tests in different files different describe names, and don't use it with test.concurrent, where tests share the current test name.

Make it faster

An eval spends most of its time waiting on the model, and Jest runs a file's tests one at a time. Jest's --shard splits by file, so one big eval file keeps the whole run waiting on its job. For a long suite, split the cases across jobs with a strategy.matrix and give each job's upload its own name. Slow evals has the details.

How evalship does this

evalship reads the eval-results.xml your workflow uploads and compares every pull request with your default branch, eval by eval. It sets aside evals that couldn't run (a missing key, a rate limit, a retired model), evals that are flaky on main, and failures within the noise it measures over each suite's last 30 runs. It reads the score and threshold you record, so a drop shows before the expect fails. Then it comments on the pull request with what's left. How the PR comment works has the details.

It also flags the prompts and LLM calls a pull request changes that no eval covers. Record targets (record({ targets: "src/support-bot.ts#answer" })) to say which code an eval exercises; see Untested changes.

With this in .evalship.yml, its check fails a pull request that breaks an eval that passed on main:

gate:
  fail_on_new_failures: true

evalship is free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart). The agent writes the workflow above for your repository.