A demo repo with sample data. Set up your own repo
Commit d65bfad on main
4 days ago · vs main@8898a24
All evals
EvalChangemainThis run
Casual tonesupport-bot / tone-casual Flaky
Input
hey whats up w my order lol
Output, same on main
Hey! It shipped yesterday and should be with you Thursday.
Threshold 0.70 · Flaky on main, so it doesn't count as a regression · History on main
Legal documentsummarizer / legal-doc Unchanged
Input
A 12-page supplier contract.
Output, same on main
The contract sets out supply terms between the two parties.
Why it failed: Judge: leaves out the termination clause and the payment terms (0.52 < 0.70)
Threshold 0.70 · History on main