Optimize prompts and models
Your coding agent starts from why your evals fail, tries changes to your prompts and models on your machine, and evalship says which win beyond chance, and by how much. You pick what ships.
37c
Instructions
Summaries that keep the terms
must-keep passes more often
To ship
- Question
- Can summaries keep a contract's termination and payment terms?
- Why this
- Legal document fails on main: the summary leaves out the termination clause and payment terms, and prompts/summarize.md:3 asks for "a short summary" without saying what it must keep.
- Answer
- must-keep passes more often, beyond chance: 41 of 42 tries, against 34 of 42 for current. Chance gives a gap this size less than 1 time in 40.
- What it means
-
For you: 8% more expensive per run.
For your users: results that pass more often.
- Next
-
A member's coding agent opens a pull request with must-keep.
currentas it is
Passed 34 / 42 tries
Cost per run as it is
Time per run as it is
must-keepBetter
3 to 32 points more often. Chance gives a gap this size less than 1 time in 40.
Passed 41 / 42 tries
Cost per run +8%
Time per run the same
How evalship tested it
Each variant ran the same 14 test cases, 3 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 40, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.
must-keep Lists what a summary keeps: parties, dates, amounts, deadlines and termination terms (prompts/summarize.md:3)
Passed more often (3)
- Long thread3/3, current 0/3
- Legal document3/3, current 0/3
- Earnings call3/3, current 2/3
bf5
Model
A cheaper model for Eliza
nano: 33% cheaper
To ship
- Question
- Can Eliza answer as well on a cheaper model?
- Follows
- More tries for 833.
- Answer
- nano is 33% cheaper and passes at most 4 points less often: 658 of 680 tries, against 669 of 680 for current. Chance gives a gap this size about 1 time in 26.
- What it means
-
For you: 33% cheaper per run.
For your users: 43% less waiting, and results as good, within 5 points.
- Next
-
A member's coding agent opens a pull request with nano.
gpt-4o-mini | gpt-4.1-nano | |
|---|---|---|
| As it is | 669 / 680 as it is | – |
| gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12) | – | Within chance658 / 680 33% cheaper · 43% less time |
currentas it is
Passed 669 / 680 tries
Cost per run as it is
Time per run as it is
nanoWithin chance
Between 4 points less often and 1 more often. Chance gives a gap this size about 1 time in 26.
Passed 658 / 680 tries
Cost per run −33%
Time per run −43%
How evalship tested it
Each variant ran the same 20 test cases, 34 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 80, so a round of many variants can't win by luck. These tries are pooled with the first look's, and a second look counts at half the bar. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.
nano gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)
Passed less often (1)
- Angry customer23/34, current 34/34
833
Model
A cheaper model for Eliza
Nothing beat current yet
- Question
- Can Eliza answer as well on a cheaper model?
- Why this
- Eliza runs gpt-4o-mini (app/support_bot.py:12), and gpt-4.1-nano costs a third less per token.
- Answer
- Nothing beat current yet: no change passed more often beyond chance, but more tries could still tell.
- Next
-
More tries in bf5.
gpt-4o-mini | gpt-4.1-nano | |
|---|---|---|
| As it is | 59 / 60 as it is | – |
| gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12) | – | Within chance58 / 60 33% cheaper · 43% less time |
currentas it is
Passed 59 / 60 tries
Cost per run as it is
Time per run as it is
nanoWithin chance
Between 10 points less often and 6 more often. Chance often gives a gap this size.
Passed 58 / 60 tries
Cost per run −33%
Time per run −43%
How evalship tested it
Each variant ran the same 20 test cases, 3 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 40, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.
nano gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)
Passed less often (1)
- Angry customer2/3, current 3/3
What it tries
40+ experiments
- ModelA newer or cheaper model, lower effort, hard inputs to a stronger one.
- ContextRetrieval, memory, tool results, a prefix your provider caches.
- InstructionsWorked examples, rules with reasons, what to do when unsure.
- OutputA schema instead of free text, fewer output tokens.
- FlowFewer or split calls, calls in parallel, code where no call is needed.
- ChecksA self-check, code for mechanical errors, judges and guardrails.
Every run is on your machine with your API keys, and your agent asks before each one, with its price. It never changes an eval to make a variant win, and a winner becomes a pull request you merge, or don't. What it never does
See what evalship finds in your evals
Ask your coding agent:
Set up https://evalship.com/setup.md