Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

Weekly report

Every Monday at 8:00 UTC, evalship emails each organization's weekly report to everyone who can see its repositories. That's the people the organization invited, and GitHub users with access to its repositories. Each person gets the repositories they can read, and only those whose evals ran or whose app sent production calls that week. An organization with neither that week gets no email. Production is in it only for the repository's members, who alone see it on evalship.

What it contains

The first line says whether anything needs you: "2 things need attention", "2 repos need attention" when more than one repository has something, or "Nothing needs you this week". The subject says the same.

Then, for each repository, the things that need someone, numbered across the email. Each says what's wrong, in the words of the flaw or the run, and has a button to where you fix it. They're what the repository's pages count in the sidebar:

  • Production issues still open, severe ones first, in To do.
  • Eval runs that stopped reaching evalship, or results it had to skip, on Test.
  • Pull requests that changed an AI call no eval tests, broke evals or left eval flaws: open ones, and ones merged that week. The email names each untested call first, then the evals that broke, and quotes each flaw; see Untested changes. On Test, each one's eval run links to its pull request, where you copy the fix for your coding agent or dismiss the flaw.
  • A review that's due, once the default branch has eval folders or suites the last review didn't read, on Test.
  • An update to your setup, when setup added something since yours, on Settings.

The email adds two more: setup, while it isn't done, and the default branch, when more of its evals fail than a week ago, not counting those its check says failed within chance. Eval flaws your team left for later aren't in it: you put them off on purpose, and Improve lists them.

Below that, a few lines, each only when there's something to act on or a catch to show:

Line When it shows
Evals on the default branch Some still fail on the latest run, beyond chance, and no numbered item says so: how many pass, and how many passed a week earlier when that differs.
Caught before merge evalship caught regressions in pull requests that a later push fixed, or that closed without merging.
Flaky evals evalship counts evals as flaky on the default branch, ignored on pull requests until they're fixed. See Flakiness.
Cost and speed The week's evals cost at least half again as much as the week before (needs cost_usd in your results; see Result formats), or pull requests wait 5 minutes or more (the median) for evals that run one at a time: their test cases' own times add up to about the workflow's. Then a pointer to Slow evals. Without test case times in your results, evalship can't tell, and leaves it out.

A repository with none of these and nothing to do stays out of the email, and an organization with none gets no email that week.

Regressions are counted the way the PR comment counts them, so evals within chance and flaky evals don't count. See How the PR comment works.

Turning it off

Each email has an unsubscribe link, and mail apps show their own unsubscribe button. Unsubscribing stops the weekly report for that person in every organization. Anyone can turn it off for themselves on Your account, and an owner can turn it off for everyone in the organization's Settings.