Moving off OpenAI Evals: promptfoo in CI without flaky reds

OpenAI's Evals dashboard and API shut down on November 30, 2026, and existing evals become read-only on October 31 (OpenAI's deprecations page). OpenAI recommends promptfoo, and its migration guide rebuilds each eval as a promptfoo config. Once an eval is a file in your repository, it can run on every pull request, and then it has to answer a question the dashboard never asked: did this pull request break it, or would it have failed anyway?

Move the evals first

Export what you can before the shutdown. In the dashboard, an eval's ⋯ menu has Export. For the eval types that support it, that gives you a runnable promptfoo config. A completed run exports as JSONL results only, which can't recreate the eval (OpenAI support, in the community forum). Rebuild the rest by hand: OpenAI's testing criteria become promptfoo assertions.

Then run each eval once and read its outputs, the passing ones too. Each should be an answer from the model, not an error page or a message about a missing key. A scorer can accept an error as an answer.

promptfoo's default gate goes red on most pull requests

promptfoo eval exits with code 100 when a single test fails, because PROMPTFOO_PASS_RATE_THRESHOLD defaults to 100% (promptfoo's command line docs). In CI, any failure fails the job.

LLM evals fail by chance. In the CI of five public repositories, a passing eval failed on the next run about 1 time in 40, and that's an upper bound (How noisy are LLM evals in CI?). At that rate:

Suite	Pass rate required	Red when the pull request changes nothing	Red when it breaks 3 evals
20 evals	100%, the default	40%	100%
50 evals	100%, the default	72%	100%
100 evals	100%, the default	92%	100%
50 evals	95%	13%	100%
50 evals	90%	0.2%	11%
100 evals	95%	4%	44%
100 evals	90%	under 0.1%	0.3%

This assumes every eval passes on main and each passing eval fails by chance 2.5% of the time, independently of the others.

A suite-wide threshold trades one mistake for the other. Set loose enough that a 100-eval suite stops going red by chance, it misses a change that breaks 3 evals more often than not. Set tight, it goes red so often that people rerun until it's green, and then a real break gets rerun too.

What a gate on promptfoo needs

Results from main on every run. Run the evals on pushes to your default branch as well as on pull requests, and upload the results even when evals fail. A pull request's results mean little without the same evals on main to compare them with.

A stable name for every test. Give each test a description. Without one, a test is known only by its vars, and editing a var turns it into a different test with no history.

Each eval against its own history. An eval that passed on every run on main and fails on a pull request means something. One that fails on main every few runs doesn't, and a suite-wide pass rate can't tell the two apart.

Errors kept apart from answers. A rate limit, a missing key or a retired model isn't a wrong answer. On PrefectHQ/prefect-mcp-server#200, 14 of 20 evals failed on the first push, all with the same 404: the judge model had been retired. Count those as couldn't run, not as failures.

A pinned judge. promptfoo picks the default grader for llm-rubric and the other model-graded assertions from whichever API keys the environment has (promptfoo's docs), so CI can grade with a different model than your laptop does. Name the judge in the config:

defaultTest:
  options:
    provider: openai:gpt-6-sol

When you change the judge, scores before and after measure different things. Compare from the change on.

Fresh answers. promptfoo caches model answers on disk for 14 days by default (caching docs). A workflow that keeps that cache between runs replays old answers instead of asking the model. Run CI with --no-cache.

Repeats for the evals that matter. --repeat 3 runs each test three times. Judge by the share that pass, so one bad sample doesn't decide. It triples the cost, so repeat the evals you gate on, not every sweep.

A workflow to start from

name: evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache -o results.json
        continue-on-error: true # the comparison with main decides, not one failure
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-results
          path: results.json

The job stays green when an eval fails, and the results of every run, on main and on pull requests, are kept as an artifact for the comparison.

How evalship does this

evalship reads the results.json your workflow uploads and compares every pull request with your default branch, eval by eval. It sets aside results that couldn't run, evals that are flaky on main, and failures within the noise it measures over each suite's last 30 runs. It counts an eval run with --repeat as passing when at least half its repeats pass, and says when a pull request changes the judge or the model, so the results before and after don't compare. Then it comments on the pull request with what's left. How PR analysis works has the details.

With this in .evalship.yml, its check fails a pull request that breaks an eval that passed on main:

gate:
  fail_on_new_failures: true