How it works

1Finds the flaws in your evals

At setup, your coding agent reads every scorer, judge and prompt, and finds where your evals fall short: evals that pass when they shouldn't, and rules no test case checks.

See the flaws in Eliza's evals →

evalship.com/gh/acme_demo/support-bot

We found 3 flaws across 44 test cases

1 CriticalEval bug

Replies that leak customer data still pass

Privacy is a fifth of the score, so a warm, helpful reply that reads out another customer's address scores 0.80 and passes.

Found in tests/evals/test_support_bot.py:44

score = 0.5 * helpfulness + 0.3 * tone + 0.2 * privacy

Fix instructions for your coding agent

Fix this flaw in our LLM evals, found by evalship's review: Replies that leak customer data still pass. Privacy is a fifth of the score, so a warm, helpful reply that reads out another customer's address scores 0.80 and passes. See tests/evals/test_support_bot.py:44. Score privacy on its own and fail the case on any leak, whatever the other scores.
Fix it in a pull request; evalship confirms the fix on it.

2 ImportantUntested instruction

Nothing checks the refund limit in the system prompt

If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give.

Found in prompts/support_system.md:12

Never promise refunds above $100

⚠ No eval tests a refund request above $100.

Fix instructions for your coding agent

Fix this flaw in our LLM evals, found by evalship's review: Nothing checks the refund limit in the system prompt. If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give. See prompts/support_system.md:12. Add cases for $150 and $1,000 refunds that expect a handoff.
Fix it in a pull request; evalship confirms the fix on it.

3 WarningEval bug

Evals pass when the judge is down

When the judge call fails, the case scores 1.0, so a run without a working judge is all green.

Found in tests/evals/conftest.py:24

return 1.0  # don't fail the build on judge errors

Fix instructions for your coding agent

Fix this flaw in our LLM evals, found by evalship's review: Evals pass when the judge is down. When the judge call fails, the case scores 1.0, so a run without a working judge is all green. See tests/evals/conftest.py:24. Fail the case on a judge error, or skip it and count the skips.
Fix it in a pull request; evalship confirms the fix on it.

2Blocks PRs that break rules

Dana's #128 shortens Eliza's prompt and drops the $100 refund limit. The refund evals your agent added after the review now fail, so evalship comments with what broke and the change that likely caused it, and its check blocks the merge.

See what broke in #128 →

github.com/acme_demo/support-bot/pull/128

evalship-appbot commented 2 minutes ago

evalship report

	Eval	main	this PR
🔴	Refund above limit	✅ 0.96	❌ 0.22
🔴	Refund far above limit	✅ 0.94	❌ 0.18

🔍 Both new failures promise refunds above the $100 limit, likely from the change to support_system.md.

Some checks were not successful

1 failing, 2 successful checks

CI / test (pull_request) Successful in 2m Details

evals / evals (pull_request) Successful in 1m Details

evalship 2 regressions in support-bot, likely from the change to support_system.md. 1 flaky eval ignored. 1 eval flaw found in this pull request. Details

Merging is blocked

The base branch requires all checks to pass.

Merge pull request

3Unblocks PRs once fixed

Dana puts the limit back. On the next push, evalship moves its comment below the new commit: the refund evals pass again, so its check passes and the pull request can merge.

How the comment moves on each push →

github.com/acme_demo/support-bot/pull/128, after the next push

dana added 1 commit: Restore the refund limit

c4d5e6f

evalship-appbot commented 2 minutes ago

evalship report

Nothing broke; 1 eval fails as on main, no eval flaws found in this pull request.

🎉 2

All checks have passed

3 successful checks

CI / test (pull_request) Successful in 2m Details

evals / evals (pull_request) Successful in 1m Details

evalship Nothing broke; 1 eval fails as on main, no eval flaws found in this pull request. Details

This branch has no conflicts with the base branch

Merging can be performed automatically.

Merge pull request

4Optimizes prompts and models

Your coding agent starts from why Eliza's evals fail, tries changes on your machine, and evalship says which win beyond chance: here, a prompt that keeps a contract's termination terms, and a model a third cheaper that passes about as often.

See what Optimize found →

evalship.com/gh/acme_demo/support-bot/optimize

37c Instructions

Summaries that keep the terms

must-keep passes more often To ship

Question

Can summaries keep a contract's termination and payment terms?

Why this

Legal document fails on main: the summary leaves out the termination clause and payment terms, and prompts/summarize.md:3 asks for "a short summary" without saying what it must keep.

Answer

must-keep passes more often, beyond chance: 41 of 42 tries, against 34 of 42 for current. Chance gives a gap this size less than 1 time in 40.

What it means

For you: 8% more expensive per run.

For your users: results that pass more often.

Next

A member's coding agent opens a pull request with must-keep.

VariantPassedCost per runTime per run

currentas it is

Passed 34 / 42 tries

Cost per run as it is

Time per run as it is

must-keepBetter

3 to 32 points more often. Chance gives a gap this size less than 1 time in 40.

Passed 41 / 42 tries

Cost per run +8%

Time per run the same

How evalship tested it

Each variant ran the same 14 test cases, 3 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 40, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.

must-keep Lists what a summary keeps: parties, dates, amounts, deadlines and termination terms (prompts/summarize.md:3)

Passed more often (3)

Long thread3/3, current 0/3
Legal document3/3, current 0/3
Earnings call3/3, current 2/3

bf5 Model

A cheaper model for Eliza

nano: 33% cheaper To ship

Question

Can Eliza answer as well on a cheaper model?

Follows

More tries for 833.

Answer

nano is 33% cheaper and passes at most 4 points less often: 658 of 680 tries, against 669 of 680 for current. Chance gives a gap this size about 1 time in 26.

What it means

For you: 33% cheaper per run.

For your users: 43% less waiting, and results as good, within 5 points.

Next

A member's coding agent opens a pull request with nano.

Models × changes
	`gpt-4o-mini`	`gpt-4.1-nano`
As it is	669 / 680 as it is	–
gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)	–	Within chance658 / 680 33% cheaper · 43% less time

VariantPassedCost per runTime per run

currentas it is

Passed 669 / 680 tries

Cost per run as it is

Time per run as it is

nanoWithin chance

Between 4 points less often and 1 more often. Chance gives a gap this size about 1 time in 26.

Passed 658 / 680 tries

Cost per run −33%

Time per run −43%

How evalship tested it

Each variant ran the same 20 test cases, 34 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 80, so a round of many variants can't win by luck. These tries are pooled with the first look's, and a second look counts at half the bar. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.

nano gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)

Passed less often (1)

Angry customer23/34, current 34/34

833 Model

A cheaper model for Eliza

Nothing beat current yet

Question: Can Eliza answer as well on a cheaper model?
Why this: Eliza runs gpt-4o-mini (app/support_bot.py:12), and gpt-4.1-nano costs a third less per token.
Answer: Nothing beat current yet: no change passed more often beyond chance, but more tries could still tell.
Next: More tries in bf5.

Models × changes
	`gpt-4o-mini`	`gpt-4.1-nano`
As it is	59 / 60 as it is	–
gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)	–	Within chance58 / 60 33% cheaper · 43% less time

VariantPassedCost per runTime per run

currentas it is

Passed 59 / 60 tries

Cost per run as it is

Time per run as it is

nanoWithin chance

Between 10 points less often and 6 more often. Chance often gives a gap this size.

Passed 58 / 60 tries

Cost per run −33%

Time per run −43%

How evalship tested it

Each variant ran the same 20 test cases, 3 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 40, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.

nano gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)

Passed less often (1)

Angry customer2/3, current 3/3

5Emails a weekly eval report

Every Monday, one email tells your team what needs them, if anything, and how the week went: here, the 2 regressions caught in #128 before they reached customers.

What's in the weekly report →

Inbox

acme_demo's evals: nothing needs you this week, 2 regressions caught before merge

evalship

To: acme_demo's team

Mon 8:00

Nothing needs you this week

acme_demo/support-bot

45 of 46 evals pass on main

Caught 2 regressions before merge
#128 Shorter support prompt · fixed

2 eval flaws left for later · In the review
Replies that leak customer data still pass
Evals pass when the judge is down
Fix them with your coding agent

1 flaky eval, ignored on pull requests until fixed: Casual tone

$34.04 on evals +19% · evals take about 4 minutes on a pull request

1Finds the flaws in your evals

Replies that leak customer data still pass

Nothing checks the refund limit in the system prompt

Evals pass when the judge is down

2Blocks PRs that break rules

evalship report

3Unblocks PRs once fixed

evalship report

4Optimizes prompts and models

Summaries that keep the terms

A cheaper model for Eliza

A cheaper model for Eliza

5Emails a weekly eval report

See what evalship finds in your evals