Pull request reports
On every pull request, evalship comments with the evals the change really broke and the likely cause, sets flaky evals and noise aside, and flags AI calls no eval covers.
See what broke in #128 →How the comment works →Block merges that break evals →
evalship report
| Eval | main | this PR | |
|---|---|---|---|
| 🔴 | Refund above limit | ✅ 0.96 | ❌ 0.22 |
| 🔴 | Refund far above limit | ✅ 0.94 | ❌ 0.18 |
🔍 Both new failures promise refunds above the $100 limit, likely from the change to support_system.md.
Some checks were not successful
1 failing, 2 successful checks
evalship
2 regressions in support-bot, likely from the change to support_system.md. 1 flaky eval ignored. 1 eval flaw found in this pull request.
Details
Merging is blocked
The base branch requires all checks to pass.
Before your agent finishes
Your coding agent runs the evals a change affects before it's done, as it runs your tests, committed or not. evalship compares them with your default branch and writes the same notes, so the pull request has its answer before you open it.
What broke, and why?
Broke
Refund above limit refund-limit-10.96 → 0.22
Refund far above limit refund-limit-20.94 → 0.18
mainthis PRpasses from 0.70
In common
🔍 Both promise refunds above the $100 limit.
Likely cause
-Never promise refunds above $100. Pass larger refunds to a colleague.
Before, after
Customer: I was charged twice for my $150 order. Can I get the full $150 back?
✓ main
Sorry about the double charge. I can refund up to $100 myself, so I've passed the remaining $50 to a colleague who will confirm within a day.
✗ this PR
So sorry about that! I've refunded the full $150 to your card. You'll see it within 3 to 5 business days.
Regression, or noise?
refund-limit-1regression
tone-casual🎲 flaky, set aside
refund-limit-1 passed every run on main, so its failure counts. tone-casual fails there now and then, so it's set aside.
Is every changed AI call tested?
+ def explain(self, order):
+ reply = client.chat.completions.create(
⚠ no eval exercises it
The pull request adds an AI call no eval runs. Its comment says so, and its report has a prompt for your agent that adds one.
See what evalship finds in your evals
Ask your coding agent:
Set up https://evalship.com/setup.md