Weekly report
Every Monday at 8:00 UTC, evalship emails each account's weekly report to everyone who reads its reports: the people the account invited, and GitHub users with access to its repositories. Each person gets the repositories they can read, and only those whose evals ran that week. An account with no eval runs that week gets no email.
What it contains
For each repository, over the 7 days before the email, it starts with the default branch's pass rate: the latest run's, and the change since the latest run a week earlier. Then one row each, most important first, every item linking to where it happened:
| Row | What it shows |
|---|---|
| Caught in pull requests | Each pull request whose evals broke that week, with how many regressions evalship found, whether a later push fixed them, and whether it's open, merged or closed. "Merged anyway" means its latest run still had them. |
| Flaws | The flaws the latest review found in your evals that are left to fix, with their severity, then those being fixed in an open pull request and those fixed. |
| Merged with no eval | LLM calls and prompts in pull requests merged that week that no eval exercises. See Untested changes. |
| Eval spend | What every eval run that week cost, pull requests included, and the change from the week before. Needs cost_usd in your results; see Result formats. |
| Flaky, set aside | The evals evalship counts as flaky on the default branch now. See Flakiness. |
Regressions are counted the way the PR comment counts them, so evals within chance and flaky evals don't count. See How PR analysis works.
Turning it off
Each email has an unsubscribe link, and mail apps show their own unsubscribe button. Unsubscribing stops the report for that person in every account. An owner of the account can turn it off for everyone on the account's page.