Case studies
Real pull requests from open-source LLM apps, replayed through evalship: what their own eval checks said, and what evalship caught.
Catches a regression from a prompt cut
HolmesGPT is an open-source SRE agent. Its #1452 commented out parts of its prompt to see which ones mattered.
HolmesGPT's eval check
Passed
It reports results but never fails the build.
evalship
Blocks the merge
162_get_runbooks fails after passing all 230 of its runs that month.
Spots a retired grading model
Prefect's MCP server gives AI assistants tools for Prefect workspaces. Its #200 fixed two of them.
Prefect's eval report
14 failed
Shown as 14 new failures against main.
evalship
14 couldn't run
The model that grades them had been retired and returned a 404, so none was graded.
See what evalship finds in your evals
Ask your coding agent:
Set up https://evalship.com/setup.md