A demo repo with sample data. Set up your own repo
37c
Instructions
Summaries that keep the terms
must-keep passes more often
To ship
- Question
- Can summaries keep a contract's termination and payment terms?
- Why this
- Legal document fails on main: the summary leaves out the termination clause and payment terms, and prompts/summarize.md:3 asks for "a short summary" without saying what it must keep.
- Answer
- must-keep passes more often, beyond chance: 41 of 42 tries, against 34 of 42 for current. Chance gives a gap this size less than 1 time in 40.
- What it means
-
For you: 8% more expensive per run.
For your users: results that pass more often.
- Next
-
A member's coding agent opens a pull request with must-keep.
currentas it is
Passed 34 / 42 tries
Cost per run as it is
Time per run as it is
must-keepBetter
3 to 32 points more often. Chance gives a gap this size less than 1 time in 40.
Passed 41 / 42 tries
Cost per run +8%
Time per run the same
How evalship tested it
Each variant ran the same 14 test cases, 3 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 40, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.
must-keep Lists what a summary keeps: parties, dates, amounts, deadlines and termination terms (prompts/summarize.md:3)
Passed more often (3)
- Long thread3/3, current 0/3
- Legal document3/3, current 0/3
- Earnings call3/3, current 2/3
bf5
Model
A cheaper model for Eliza
nano: 33% cheaper
To ship
- Question
- Can Eliza answer as well on a cheaper model?
- Follows
- More tries for 833.
- Answer
- nano is 33% cheaper and passes at most 4 points less often: 658 of 680 tries, against 669 of 680 for current. Chance gives a gap this size about 1 time in 26.
- What it means
-
For you: 33% cheaper per run.
For your users: 43% less waiting, and results as good, within 5 points.
- Next
-
A member's coding agent opens a pull request with nano.
gpt-4o-mini | gpt-4.1-nano | |
|---|---|---|
| As it is | 669 / 680 as it is | – |
| gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12) | – | Within chance658 / 680 33% cheaper · 43% less time |
currentas it is
Passed 669 / 680 tries
Cost per run as it is
Time per run as it is
nanoWithin chance
Between 4 points less often and 1 more often. Chance gives a gap this size about 1 time in 26.
Passed 658 / 680 tries
Cost per run −33%
Time per run −43%
How evalship tested it
Each variant ran the same 20 test cases, 34 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 80, so a round of many variants can't win by luck. These tries are pooled with the first look's, and a second look counts at half the bar. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.
nano gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)
Passed less often (1)
- Angry customer23/34, current 34/34
833
Model
A cheaper model for Eliza
Nothing beat current yet
- Question
- Can Eliza answer as well on a cheaper model?
- Why this
- Eliza runs gpt-4o-mini (app/support_bot.py:12), and gpt-4.1-nano costs a third less per token.
- Answer
- Nothing beat current yet: no change passed more often beyond chance, but more tries could still tell.
- Next
-
More tries in bf5.
gpt-4o-mini | gpt-4.1-nano | |
|---|---|---|
| As it is | 59 / 60 as it is | – |
| gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12) | – | Within chance58 / 60 33% cheaper · 43% less time |
currentas it is
Passed 59 / 60 tries
Cost per run as it is
Time per run as it is
nanoWithin chance
Between 10 points less often and 6 more often. Chance often gives a gap this size.
Passed 58 / 60 tries
Cost per run −33%
Time per run −43%
How evalship tested it
Each variant ran the same 20 test cases, 3 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 40, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.
nano gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)
Passed less often (1)
- Angry customer2/3, current 3/3
Also proposed, not tried: Fewer retrieved help articles for Eliza (context) · A JSON schema for the extractor's reply (output)