Connecting CI

evalship reads eval results from GitHub Actions workflow artifacts. When a workflow run in your repository completes, GitHub sends the evalship App a workflow_run event with the action completed, and evalship reads the result files in that run's artifacts.

There is no evalship action to add, and no token or secret to configure.

Requirements

1. The eval workflow runs on pull_request and on push to the default branch. Pull request runs get the comment. Default branch runs are the baseline that pull requests are compared against; without them, every pull request looks like a first run. Setup's workflow puts a paths filter on pull_request, so evals run only on pull requests that change what they test; on a repository with a review, one it skips still gets a comment when it changes a prompt or AI call. To run them on every pull request, delete paths. A branch rule that requires the eval job waits forever on a pull request the filter skips, so require the evalship check instead. If the workflow has a concurrency group, cancel only pull request runs (cancel-in-progress: ${{ github.event_name == 'pull_request' }}), so no push to the default branch loses its results.
2. The eval command writes a results file: JUnit XML, promptfoo JSON, pytest-json-report JSON, Vitest or Jest JSON, an Inspect AI log or evalship JSON. See Result formats.
3. An actions/upload-artifact@v7 step with if: always() uploads the file. Without if: always() the upload is skipped when an eval fails, which is the run you most want reported.

      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Which check decides

A workflow evalship's setup creates gives its eval step continue-on-error: true. The job stays green when an eval fails or flips, and evalship's evalship check decides whether a pull request broke the evals: it compares with the default branch and leaves out evals it has seen flip. To have the eval job itself fail instead, remove that line. A workflow you already had keeps its exit status.

Artifact names

The name evalship-results is recommended, not required. By default evalship scans every artifact of every completed workflow run for result files, so a workflow that already uploads JUnit XML needs no change.

To read only some artifacts, list name globs in .evalship.yml:

results:
  artifacts:
    - evalship-results
    - "evals-*"

What evalship reads

.xml, .json and .eval files that aren't eval results, such as package.json or a coverage report, are skipped. If an artifact whose name contains eval, junit, result, test or promptfoo holds no readable result file, the dashboard shows "Eval runs aren't reaching evalship." with the workflow and artifact names.

To upload several files, list them or upload a directory:

        with:
          name: evalship-results
          path: |
            eval-results.xml
            promptfoo-results.json

Examples

Add the env your evals need.

pytest

      - run: pytest tests/evals --junitxml=eval-results.xml -o junit_family=xunit1
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

-o junit_family=xunit1 keeps record_property values in the XML, which is how scores and thresholds reach evalship. See Result formats.

promptfoo

      - run: npx promptfoo@latest eval -c promptfooconfig.yaml -o eval-results.json
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.json

Vitest

      - run: npx vitest run evals --reporter=junit --outputFile=eval-results.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Jest

Install jest-junit as a dev dependency.

      - run: npx jest evals --ci --reporters=default --reporters=jest-junit
        env:
          JEST_JUNIT_OUTPUT_FILE: eval-results.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

RSpec

Add gem "rspec_junit_formatter" to the test group of your Gemfile.

      - run: bundle exec rspec spec/evals --format progress --format RspecJunitFormatter --out eval-results.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Go

      - run: go install github.com/jstemmer/go-junit-report/v2@latest
      - run: go test -v ./evals/... 2>&1 | go-junit-report -set-exit-code > eval-results.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Java

Maven writes a JUnit XML report for every test class without extra setup:

      - run: mvn -B test -Dtest='*EvalTest'
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: target/surefire-reports/*.xml

With Gradle, run ./gradlew test and upload build/test-results/test/*.xml. Kotlin tests write the same reports.

PHP

      - run: vendor/bin/phpunit --log-junit eval-results.xml tests/Evals
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Rust

cargo test can't write JUnit XML on stable Rust, so run the evals with cargo-nextest and give it a report path in .config/nextest.toml:

[profile.ci.junit]
path = "junit.xml"

      - uses: taiki-e/install-action@nextest
      - run: cargo nextest run --profile ci --test evals
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: target/nextest/ci/junit.xml

nextest's retries count once: a test that passes on a retry passes.

tinyeval

tinyeval runs evals from one bash file the repository commits, in any language, and writes evalship JSON:

      - run: ./tinyeval -n 3 -j 4
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.json

-n 3 runs each test case 3 times, and evalship reads the repeats as one eval.

Shell script

Evals can be a bash script with no framework at all. This one asks a command-line bot each question, checks the reply with grep and writes evalship JSON with jq, which GitHub's Ubuntu runners include:

#!/usr/bin/env bash
# evals/run.sh: each case asks the bot one question and checks its reply with a pattern.
evals='[]'
check() { # key, question, pattern the reply must match
  local reply passed=false
  reply=$(./bin/support-bot "$2")
  grep -qiE "$3" <<<"$reply" && passed=true
  evals=$(jq -c --arg key "$1" --arg input "$2" --arg output "$reply" --argjson passed "$passed" \
    '. + [{key: $key, input: $input, output: $output, passed: $passed}]' <<<"$evals")
}
check refund-above-limit 'Can I get a refund of $500?' 'colleague'
check order-status 'Where is order 1042?' '1042'
jq -n --argjson evals "$evals" '{suite: "support-bot", evals: $evals}' > eval-results.json

      - run: bash evals/run.sh
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.json

Full workflow

A complete .github/workflows/evals.yml for a Python repository with main as the default branch:

name: evals

on:
  pull_request:
    paths:
      - "tests/evals/**"
      - "app/llm/**"
      - "prompts/**"
      - "requirements.txt"
      - ".github/workflows/evals.yml"
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/evals --junitxml=eval-results.xml -o junit_family=xunit1
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Several workflows or artifacts

All artifacts from all workflow runs for the same commit SHA merge into one run. evalship processes it about 15 seconds after the latest artifact arrives; if another workflow finishes later, its results are added and the comment is updated. When two artifacts report the same suite and key, the one read last wins.

So slow evals can run in parallel jobs, as the next section says. The 20,000-eval limit counts every artifact in the workflow run together.

Slow evals

An eval run spends most of its time waiting on the model, and test runners run cases one at a time by default. So first run them concurrently in the same job: pytest-xdist -n 4, promptfoo --max-concurrency, and Promise.all or asyncio.gather for a case's repeats. Your provider's rate limit sets how far this goes. Vitest's concurrent tests put a test's printed output under another case in the JUnit report, so there, split cases across parallel jobs instead.

If a run still takes more than 5 minutes, split the cases across a strategy.matrix and give each job's upload its own name, such as evalship-results-${{ matrix.shard }}. A 16-minute job on 4 shards takes about 4 minutes, for the same number of model calls. Split by case, not by file: vitest's and jest's --shard split by file, so one big file keeps the whole run waiting on its job. Check that the jobs together run every case once.

Repeats multiply a run's time. Run a case 3 times only where it flips and variety is the point, such as a tweet: evalship lists those as flaky and sets them aside on pull requests (see Checks).

A push run on a branch with an open pull request counts for that pull request when the commit is the pull request's head.

Run evals when you want

Setup's workflow also has a workflow_dispatch trigger, so you can run the evals on a branch once it's ready: in the Actions tab under Run workflow, or with gh workflow run evals.yml --ref BRANCH. The results go to the pull request whose head is that commit, as a push's do. To run them only then, delete the pull_request trigger and keep push to the default branch for the baseline.

Run evals on your machine

To keep your model keys off GitHub, run the evals yourself and send the results file. The first time, your coding agent opens an evalship page where you approve it, and keeps its upload token in ~/.config/evalship/token. Without an agent, make a token in Settings and save it as EVALSHIP_TOKEN. Push the commit, then from the repository:

curl --fail-with-body -sS https://evalship.com/api/v1/results \
  -H "Authorization: Bearer $EVALSHIP_TOKEN" \
  -F repository=OWNER/REPO \
  -F sha=$(git rev-parse HEAD) \
  -F branch=$(git branch --show-current) \
  -F file=@eval-results.xml

Send each results file in its own request. evalship treats them as CI's: a default branch commit's results are the baseline, and a pull request's head commit gets the check and comment, which say who ran them. Run them on a clean working tree, since the results count for the commit.

Your coding agent can send the default branch's first: ask it Connect this machine and run evalship's evals for the default branch: https://evalship.com/pr.md?repo=OWNER/REPO. Setup offers the same when your evals need a key the repository doesn't have. From then on the repository runs its evals on your machines, until a member picks Use GitHub Actions instead on its Settings page. A pull request that changes prompts, AI calls or evals gets a comment right after each push with one line to give your coding agent. The agent checks out the pull request, runs the evals the way your workflow does and sends the results, and the comment becomes the report.

There your coding agent writes the pull request's notes too, on your own model plan, so no excerpt of the change goes to evalship's model. It gets the instructions this change needs, writes the notes on its checkout and sends them; evalship drops any it can't check against the diff, as it does its own model's, and the comment says whose agent wrote them. Then the agent tells you what the pull request's check says, such as "2 regressions in support-bot", and offers to fix them on the branch.

CI's default branch results still count, so pull requests may compare against a run without your keys. Settings warns when CI sends them: run the eval workflow only on: workflow_dispatch.

A token sends results to every repository you can push to on GitHub. Settings lists yours, with when each was last used; revoking one stops it.

Before you finish

Your coding agent can run the evals a change affects before it's done, as it runs the tests, committed or not. Put this line in the repository's AGENTS.md, CLAUDE.md or other agent instructions. Setup adds it, and creates an AGENTS.md when the repository has none, unless you say no:

In OWNER/REPO, before you finish a change to a prompt, a model or an LLM call, follow https://evalship.com/change.md?repo=OWNER/REPO, which sends evalship the change's diff with its results

The agent sends the change's diff from your machine, and evalship answers which suites it affects, so it runs only those, or none. Its results are compared with the default branch, and it writes the notes. Its run says whose machine it came from, and evalship deletes the diff after 7 days, or once the commit reaches GitHub. When you commit and push the same code, the pull request takes that run, with its report and notes, and nothing runs again. Pushed straight to the default branch, it becomes that commit's run when your agent checks its next change, if it ran every suite. This works whether the repository's evals run in GitHub Actions or on your machines.

A repository set up before this gets it from Update your setup on its Settings page, which gives your coding agent one line for every change setup has made since.

API keys

Your evals call your model provider with your own key, saved as an Actions secret. If they call Claude, a Claude Max, Ultra or Team plan includes $100 to $500 a month of API credit: link a Console organization in claude.ai under Settings > Billing and use a key from that organization (how, and what a run costs). When the credit runs out and the organization has no other, calls stop until next month's arrives, and evalship shows those evals as couldn't run, not as failures.

Fork pull requests

Fork pull requests work without extra setup. GitHub doesn't pass your Actions secrets to workflows triggered from forks or by Dependabot. If your evals call a paid API, they fail on those pull requests and evalship would report those failures as regressions. To skip evals there, add a condition to the job:

jobs:
  evals:
    if: github.event_name != 'pull_request' || (github.event.pull_request.head.repo.full_name == github.repository && github.event.pull_request.user.login != 'dependabot[bot]')

Those pull requests then get no results, and a comment only if they change prompts or AI calls, saying no eval ran.

evalship skips results from pull_request_target workflows. GitHub files those runs under your default branch's latest commit, so a fork's results would read as your default branch's.

Timing and retention

evalship downloads artifacts when the completed event arrives, usually within seconds of the run finishing, and stores the parsed results. Your repository's artifact retention setting doesn't matter after that. Artifacts that have already expired when the event arrives are skipped.

Re-running the workflow uploads new artifacts and updates the results for the commit. Re-running the evalship check from GitHub re-processes the stored results without downloading anything.

Other CI systems

Only GitHub Actions is supported for now. The results have to be a workflow artifact in the same repository.

Troubleshooting