Connecting CI

evalship reads eval results from GitHub Actions workflow artifacts. When a workflow run in your repository completes, GitHub sends the evalship App a workflow_run event with the action completed. evalship downloads that run's artifacts, parses the result files inside, and processes the results for the commit.

There is no evalship action to add, and no token or secret to configure.

Requirements

1. The eval workflow runs on pull_request and on push to the default branch. Pull request runs get the comment. Default branch runs are the baseline that pull requests are compared against; without them, every pull request looks like a first run.
2. The eval command writes a results file: JUnit XML, promptfoo JSON or evalship JSON. See Result formats.
3. An actions/upload-artifact@v4 step with if: always() uploads the file. Without if: always() the upload is skipped when an eval fails, which is the run you most want reported.

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Artifact names

The name evalship-results is recommended, not required. By default evalship scans every artifact of every completed workflow run for result files, so a workflow that already uploads JUnit XML needs no change.

To read only some artifacts, list name globs in .evalship.yml:

results:
  artifacts:
    - evalship-results
    - "evals-*"

What evalship reads

.xml and .json files that aren't eval results, such as package.json or a coverage report, are skipped. If an artifact whose name contains eval, junit, result, test or promptfoo holds no readable result file, the report page shows "Results aren't reaching evalship" with the workflow and artifact names.

To upload several files, list them or upload a directory:

        with:
          name: evalship-results
          path: |
            eval-results.xml
            promptfoo-results.json

Examples

Each example writes a results file and uploads it. Add the env your evals need.

pytest

      - run: pytest tests/evals --junitxml=eval-results.xml -o junit_family=xunit1
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

-o junit_family=xunit1 keeps record_property values in the XML, which is how scores and thresholds reach evalship. See Result formats.

promptfoo

      - run: npx promptfoo@latest eval -c promptfooconfig.yaml -o eval-results.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalship-results
          path: eval-results.json

Vitest

      - run: npx vitest run evals --reporter=junit --outputFile=eval-results.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Jest

Install jest-junit as a dev dependency.

      - run: npx jest evals --ci --reporters=default --reporters=jest-junit
        env:
          JEST_JUNIT_OUTPUT_FILE: eval-results.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

RSpec

Add gem "rspec_junit_formatter" to the test group of your Gemfile.

      - run: bundle exec rspec spec/evals --format progress --format RspecJunitFormatter --out eval-results.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Go

      - run: go install github.com/jstemmer/go-junit-report/v2@latest
      - run: go test -v ./evals/... 2>&1 | go-junit-report -set-exit-code > eval-results.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Full workflow

A complete .github/workflows/evals.yml for a Python repository with main as the default branch:

name: evals

on:
  pull_request:
  push:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/evals --junitxml=eval-results.xml -o junit_family=xunit1
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

Several workflows or artifacts

Results are grouped by commit. All artifacts from all workflow runs for the same commit SHA merge into one run. evalship processes it about 15 seconds after the latest artifact arrives; if another workflow finishes later, its results are added and the comment is updated. When two artifacts report the same suite and key, the one read last wins.

A push run on a branch with an open pull request counts for that pull request when the commit is the pull request's head.

Fork pull requests

Fork pull requests work without extra setup. The pull_request workflow in the fork's context uploads the artifact, which needs no secret, and the evalship App reads it from your repository.

GitHub doesn't pass secrets to workflows triggered from forks. If your evals call a paid API, they fail on fork pull requests and evalship would report those failures as regressions. To skip evals on forks, add a condition to the job:

jobs:
  evals:
    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository

Fork pull requests then get no results and no comment.

Timing and retention

evalship downloads artifacts when the completed event arrives, usually within seconds of the run finishing, and stores the parsed results. Your repository's artifact retention setting doesn't matter after that. Artifacts that have already expired when the event arrives are skipped.

Re-running the workflow uploads new artifacts and updates the results for the commit. Re-running the evalship check from GitHub re-processes the stored results without downloading anything.

Other CI systems

Only GitHub Actions is supported for now. The results have to be a workflow artifact in the same repository.

Troubleshooting