A demo repo with sample data. Set up your own repo
Commit 475a178 on main
24 days ago · vs main@3499290
All evals
Legal documentsummarizer / legal-doc Unchanged
Input
A 12-page supplier contract.
Output, same on main
The contract sets out supply terms between the two parties.
Why it failed: Judge: leaves out the termination clause and the payment terms (0.52 < 0.70)
Threshold 0.70 · History on main
Currency codeextractor / currency-code Unchanged
Input
Invoice INV-3310 (PDF, in euros).
Output, same on main
{"currency": "EUR"}
Threshold 0.70 · History on main
Due dateextractor / due-date Unchanged
Input
Invoice INV-2207 (PDF).
Output, same on main
{"due_date": "2026-06-13"}
Threshold 0.70 · History on main
Empty inputextractor / empty-input Unchanged
Input
An empty PDF.
Output, same on main
{"error": "no invoice found"}
Threshold 0.70 · History on main
Invoice dateextractor / invoice-date Unchanged
Input
Invoice INV-2207 (PDF).
Output, same on main
{"date": "2026-05-14"}
Threshold 0.70 · History on main
Invoice totalextractor / invoice-total Unchanged
Input
Invoice INV-2207 (PDF).
Output, same on main
{"total": 1249.00}
Threshold 0.70 · History on main
JSON shapeextractor / json-shape Unchanged
Input
Invoice INV-2207 (PDF).
Output, same on main
{"total": 1249.00, "date": "2026-05-14", "vendor": "Northwind Supplies"}
Threshold 0.70 · History on main
Line itemsextractor / line-items Unchanged
Input
Invoice INV-2207 (PDF).
Output, same on main
{"line_items": [{"sku": "MUG-01", "qty": 40}, {"sku": "BOX-12", "qty": 3}]}
Threshold 0.70 · History on main
PO numberextractor / po-number Unchanged
Input
Invoice INV-2207 (PDF).
Output, same on main
{"po_number": "PO-5521"}
Threshold 0.70 · History on main
Receipt totalextractor / receipt-total Unchanged
Input
A photo of a restaurant receipt.
Output, same on main
{"total": 86.40}
Threshold 0.70 · History on main
Rotated scanextractor / rotated-scan Unchanged
Input
Invoice INV-1984, scanned upside down.
Output, same on main
{"total": 310.50}
Threshold 0.70 · History on main
Tax IDextractor / tax-id Unchanged
Input
Invoice INV-3310 (PDF, in euros).
Output, same on main
{"tax_id": "NL001234567B01"}
Threshold 0.70 · History on main
Vendor nameextractor / vendor-name Unchanged
Input
Invoice INV-2207 (PDF).
Output, same on main
{"vendor": "Northwind Supplies"}
Threshold 0.70 · History on main
Bullet pointssummarizer / bullet-points Unchanged
Input
A product spec, summarized as bullet points.
Output, same on main
- Bulk edit for orders - Undo for 30 seconds - Ships behind a flag
Threshold 0.70 · History on main
Changelogsummarizer / changelog Unchanged
Input
The changelog for version 2.4.
Output, same on main
Version 2.4 adds CSV export and fixes the timezone bug in reports.
Threshold 0.70 · History on main
Earnings callsummarizer / earnings-call Unchanged
Input
A transcript of a quarterly earnings call.
Output, same on main
Revenue grew 18% to $42M. Margins fell 2 points on shipping costs. Guidance is unchanged.
Threshold 0.70 · History on main
Email threadsummarizer / email-thread Unchanged
Input
An 8-email thread about an invoice dispute.
Output, same on main
The vendor double-billed March. They agreed to credit $1,240 on the next invoice.
Threshold 0.70 · History on main
Long threadsummarizer / long-thread Unchanged
Input
A 42-message support thread about a delayed order.
Output, same on main
The customer's order was delayed, and the agent offered a refund.
Threshold 0.70 · History on main
Meeting notessummarizer / meeting-notes Unchanged
Input
Notes from the weekly product meeting.
Output, same on main
Launch moves to June 4. Priya owns the pricing page, and Sam the launch email.
Threshold 0.70 · History on main
News articlesummarizer / news-article Unchanged
Input
A news article about a port strike.
Output, same on main
A strike at the port of Rotterdam delays container shipments by about a week.
Threshold 0.70 · History on main
PDF reportsummarizer / pdf-report Unchanged
Input
A 30-page quarterly support report.
Output, same on main
Tickets rose 12%, mostly about shipping. First-reply time held at 2 hours.
Threshold 0.70 · History on main
Press releasesummarizer / press-release Unchanged
Input
A press release about an acquisition.
Output, same on main
Acme buys Brightline, a returns startup, for an undisclosed sum. The deal closes in July.
Threshold 0.70 · History on main
Research papersummarizer / research-paper Unchanged
Input
A 9-page paper on retrieval methods.
Output, same on main
Hybrid search beats dense-only retrieval on 4 of 5 benchmarks, at twice the latency.
Threshold 0.70 · History on main
Short articlesummarizer / short-article Unchanged
Input
A 300-word blog post about a product update.
Output, same on main
Saved searches now sync across devices, starting today.
Threshold 0.70 · History on main
Slack threadsummarizer / slack-thread Unchanged
Input
A Slack thread about a failed deploy.
Output, same on main
Friday's deploy failed on a missing migration. It was rolled back and fixed on Monday.
Threshold 0.70 · History on main
Call transcriptsummarizer / transcript Unchanged
Input
A 20-minute customer call transcript.
Output, same on main
The customer wants to upgrade to the team plan but needs SSO first.
Threshold 0.70 · History on main
Angry customersupport-bot / angry-customer Unchanged
Input
This is the third time my order is late. Unacceptable.
Output, same on main
You're right, that's not good enough. I've upgraded your shipping for free and flagged the delays.
Threshold 0.70 · History on main
Cancel ordersupport-bot / cancel-order Unchanged
Input
Please cancel the order I placed this morning.
Output, same on main
Done: this morning's order is cancelled, and you won't be charged.
Threshold 0.70 · History on main
Change addresssupport-bot / change-address Unchanged
Input
Can I still change the delivery address?
Output, same on main
Yes, until it ships. What's the new address?
Threshold 0.70 · History on main
Damaged itemsupport-bot / damaged-item Unchanged
Input
The mug arrived cracked.
Output, same on main
Sorry! I'm sending a new one today. No need to return the cracked one.
Threshold 0.70 · History on main
Discount codesupport-bot / discount-code Unchanged
Input
My code SPRING10 doesn't work.
Output, same on main
SPRING10 ended on May 31, so I've added 10% off to your next order instead.
Threshold 0.70 · History on main
Escalationsupport-bot / escalation Unchanged
Input
I want to talk to a human.
Output, same on main
Of course. I'm passing you to a colleague now. The wait is about 5 minutes.
Threshold 0.70 · History on main
Gift cardsupport-bot / gift-card Unchanged
Input
Can I pay with a gift card and a credit card?
Output, same on main
Yes. Enter the gift card first and pay the rest by card.
Threshold 0.70 · History on main
Greetingsupport-bot / greeting Unchanged
Human handoffsupport-bot / human-handoff Unchanged
Input
Can someone call me back?
Output, same on main
Yes, a colleague will call you within one business day. Which number should they use?
Threshold 0.70 · History on main
Lost packagesupport-bot / lost-package Unchanged
Input
Tracking says delivered but nothing's here.
Output, same on main
Sorry about that. I've opened a trace with the carrier and will update you within 48 hours.
Threshold 0.70 · History on main
Order statussupport-bot / order-status Unchanged
Input
Where is order 18234?
Output, same on main
Order 18234 shipped on Monday and should arrive Thursday.
Threshold 0.70 · History on main
Privacy requestsupport-bot / privacy-request Unchanged
Input
Delete all my data please.
Output, same on main
I've sent your deletion request to our privacy team. You'll get a confirmation by email within 30 days.
Threshold 0.70 · History on main
Refund under limitsupport-bot / refund-small Unchanged
Input
My $40 order never arrived. Can I get a refund?
Output, same on main
I'm sorry it never arrived. I've refunded the $40 to your card.
Threshold 0.70 · History on main
Return labelsupport-bot / return-label Unchanged
Input
How do I return these shoes?
Output, same on main
I've emailed you a prepaid return label. Drop the parcel at any post office.
Threshold 0.70 · History on main
Shipping to the EUsupport-bot / shipping-eu Unchanged
Input
Do you ship to Germany?
Output, same on main
Yes, to every EU country. Delivery takes 3 to 5 business days.
Threshold 0.70 · History on main
Shipping in the USsupport-bot / shipping-us Unchanged
Input
How long does shipping to Texas take?
Output, same on main
Standard shipping takes 2 to 4 business days.
Threshold 0.70 · History on main
Sizing helpsupport-bot / sizing-help Unchanged
Input
I'm usually a medium. Which size for the rain jacket?
Output, same on main
It runs large, so a small should fit you.
Threshold 0.70 · History on main
Casual tonesupport-bot / tone-casual Unchanged
Input
hey whats up w my order lol
Output, same on main
Hey! It shipped yesterday and should be with you Thursday.
Threshold 0.70 · History on main