Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

Configuration

evalship works without configuration. To change its behavior, add .evalship.yml to the root of your repository.

evalship reads the file at the commit it is processing, with one exception: a pull request is judged by the gate and comparison of the branch it merges into, so it can't loosen its own check. A change to those takes effect once it merges. Its comment and results settings apply to the pull request right away.

All keys with their defaults

comparison:
  tolerance: 0.05
  suites: {}

gate:
  fail_on_new_failures: false
  fail_on_flaws: false
  max_pass_rate_drop: null
  ignore_flaky: true

comment:
  enabled: true
  only_on_change: false
  repost: true

results:
  artifacts: []

monitor: {}

comparison

Key Type Default Effect
tolerance number 0.05 A score change counts as regressed or improved only when the absolute difference is larger than this. Also used for flakiness: an eval whose scores vary with a standard deviation above twice the tolerance is flaky.
suites map {} Settings per suite name. Each suite accepts tolerance (number) and required (true or false).

A suite with required: true fails the check when it has no results for a commit, for example because its workflow didn't run. Setting it also turns the check from advisory into a gate (see below). A workflow with a paths filter, as setup writes, doesn't run on every pull request, so leave its suites out of required.

comparison:
  tolerance: 0.05
  suites:
    support_bot:
      tolerance: 0.1
      required: true

Suite names are the names in the comment's suite table. Result formats explains how each format derives them.

gate

The evalship check is neutral unless at least one of these is set: fail_on_new_failures: true, fail_on_flaws: true, a number for max_pass_rate_drop, or required: true on a suite. Once one is set, the check concludes success or failure.

Key Type Default Effect
fail_on_new_failures boolean false Fail the check when a non-flaky eval went from passing to failing, beyond chance, on a pull request that changes app code. In a suite the default branch has run fewer than 3 times, a failure could still be chance, so it fails the check only when it also failed on the pull request's previous push. One failing eval that never fails on the default branch can still be chance next to the other evals, so evalship reruns its workflow once (its failed jobs, or the whole workflow when none failed) and the check waits for that run. It also fails when evals that ran on the baseline couldn't run on a pull request that had the repository's secrets; forks and Dependabot get none, so theirs don't count, and neither do provider outages.
fail_on_flaws boolean false Fail the check while the pull request has an eval flaw nobody fixed or dismissed: a note from the PR analysis, or an AI call no eval tests. Dismissing it in the report (Wrong, Won't fix, Fixed or Later) clears it, so merging with a flaw is a member's decision. It also fails when the pull request changes an AI call or prompt whose suites run on pull requests and no results came for the commit. While the PR analysis runs, the check stays in progress.
max_pass_rate_drop number or null null Fail the check when the pass rate dropped by more than this many percentage points against the baseline. 2 allows 95.8% to 93.8%, not lower.
ignore_flaky boolean true Leave flaky evals out of regressions, improvements and the gate; max_pass_rate_drop is then measured on the non-flaky evals only. false counts flaky evals like any other, so their flips can be regressions and fail the check. Either way, failures within chance never do.

See How the PR comment works for baselines and flakiness.

comment

Key Type Default Effect
enabled boolean true false stops PR comments, as the PR comments switch on Settings does. The evalship check is still posted.
only_on_change boolean false true skips the comment when nothing changed against the baseline. A pull request that already has an evalship comment still gets it updated.
repost boolean true Each new commit gets its report in a new comment at the bottom of the pull request, below that commit, and the one before is hidden as outdated. The same commit reported again edits its comment. false edits the first comment in place, which sends no notification for later commits.

results

Key Type Default Effect
artifacts list of strings [] Artifact name globs to read results from, such as evalship-results or evals-*. Empty means every artifact is scanned for result files.

monitor

Checks for your production AI calls, by the call's name: a list each, such as json, contains: "text" or judge: "a yes-or-no question". A call you list gets only its checks, plus the reply checks if you list replies; a call you don't list gets the reply checks. Monitor has every check. evalship reads them from the default branch.

monitor:
  classify-ticket:
    - json
    - matches: '"category": "(billing|bug|other)"'

Validation

Problem What happens
Unknown section or key Ignored.
Wrong type, such as tolerance: "high" The default is used for that key.
Invalid entry under comparison.suites That entry is dropped; valid entries are kept.
File is not a mapping, or not valid YAML All defaults are used.

Problems are listed in the evalship check output under ".evalship.yml has problems, so defaults were used for these keys". The repository's settings on evalship show the same list and the effective configuration of the latest run.

Examples

Block pull requests that break evals, ignoring flaky ones:

gate:
  fail_on_new_failures: true

Allow small pass rate dips, but always require the safety suite:

gate:
  max_pass_rate_drop: 2
comparison:
  suites:
    safety:
      required: true

Comment only when something changed, and read one artifact:

comment:
  only_on_change: true
results:
  artifacts: [evalship-results]

A check that blocks merging also needs a branch protection rule or ruleset on GitHub that requires the evalship check. GitHub offers those on a private repository only with GitHub Pro (Team for an organization). A neutral check never blocks a merge.