Configuration

evalship works without configuration. To change its behavior, add .evalship.yml to the root of your repository.

evalship reads the file at the commit it is processing. A pull request that changes .evalship.yml is evaluated with its own version of the file.

All keys with their defaults

comparison:
  tolerance: 0.05
  suites: {}

gate:
  fail_on_new_failures: false
  max_pass_rate_drop: null
  ignore_flaky: true

comment:
  enabled: true
  only_on_change: false

results:
  artifacts: []

comparison

Key	Type	Default	Effect
`tolerance`	number	`0.05`	A score change counts as regressed or improved only when the absolute difference is larger than this. Also used for flakiness: a case whose scores vary with a standard deviation above twice the tolerance is flaky.
`suites`	map	`{}`	Settings per suite name. Each suite accepts `tolerance` (number) and `required` (`true` or `false`).

A suite with required: true fails the check when it has no results for a commit, for example because its workflow didn't run. Setting it also turns the check from advisory into a gate (see below).

comparison:
  tolerance: 0.05
  suites:
    support_bot:
      tolerance: 0.1
      required: true

Suite names are the names in the comment's suite table. Result formats explains how each format derives them.

gate

The evalship check is neutral unless at least one of these is set: fail_on_new_failures: true, a number for max_pass_rate_drop, or required: true on a suite. Once one is set, the check concludes success or failure.

Key	Type	Default	Effect
`fail_on_new_failures`	boolean	`false`	Fail the check when any eval went from passing to failing against the baseline.
`max_pass_rate_drop`	number or `null`	`null`	Fail the check when the pass rate dropped by more than this many percentage points against the baseline. `2` allows 95.8% to 93.8%, not lower.
`ignore_flaky`	boolean	`true`	Detect flaky evals and leave them out of regressions, improvements and the gate. When flaky evals are involved, `max_pass_rate_drop` is measured on the non-flaky evals only. `false` turns flakiness detection off, so every flip counts.

See How PR analysis works for baselines and flakiness.

comment

Key	Type	Default	Effect
`enabled`	boolean	`true`	`false` stops PR comments. The `evalship` check is still posted.
`only_on_change`	boolean	`false`	`true` skips the comment when nothing changed against the baseline. A pull request that already has an evalship comment still gets it updated.

results

Key	Type	Default	Effect
`artifacts`	list of strings	`[]`	Artifact name globs to read results from, such as `evalship-results` or `evals-*`. Empty means every artifact is scanned for result files.

Validation

Mistakes don't stop evalship; it falls back to defaults and tells you.

Problem	What happens
Unknown section or key	Ignored.
Wrong type, such as `tolerance: "high"`	The default is used for that key.
Invalid entry under `comparison.suites`	That entry is dropped; valid entries are kept.
File is not a mapping, or not valid YAML	All defaults are used.

Problems are listed in the evalship check output under ".evalship.yml has problems, so defaults were used for these keys". The repository's settings page on evalship shows the same list and the effective configuration of the latest run.

Examples

Block pull requests that break evals, ignoring flaky ones:

gate:
  fail_on_new_failures: true

Allow small pass rate dips, but always require the safety suite:

gate:
  max_pass_rate_drop: 2
comparison:
  suites:
    safety:
      required: true

Comment only when something changed, and read one artifact:

comment:
  only_on_change: true
results:
  artifacts: [evalship-results]

A check that blocks merging also needs a branch protection rule or ruleset on GitHub that requires the evalship check. A neutral check never blocks a merge.