A demo repo with sample data. Set up your own repo
Commit 1cdd967 on main
about 1 month ago
All evals
Legal documentsummarizer / legal-doc Unchanged
Input
A 12-page supplier contract.
Output
The contract sets out supply terms between the two parties.
Why it failed: Judge: leaves out the termination clause and the payment terms (0.52 < 0.70)
Threshold 0.70 · History on main
Currency codeextractor / currency-code Unchanged
Due dateextractor / due-date Unchanged
Empty inputextractor / empty-input Unchanged
Invoice dateextractor / invoice-date Unchanged
Invoice totalextractor / invoice-total Unchanged
JSON shapeextractor / json-shape Unchanged
Input
Invoice INV-2207 (PDF).
Output
{"total": 1249.00, "date": "2026-05-14", "vendor": "Northwind Supplies"}
Threshold 0.70 · History on main
Line itemsextractor / line-items Unchanged
Input
Invoice INV-2207 (PDF).
Output
{"line_items": [{"sku": "MUG-01", "qty": 40}, {"sku": "BOX-12", "qty": 3}]}
Threshold 0.70 · History on main
PO numberextractor / po-number Unchanged
Receipt totalextractor / receipt-total Unchanged
Rotated scanextractor / rotated-scan Unchanged
Input
Invoice INV-1984, scanned upside down.
Output
{"total": 310.50}
Threshold 0.70 · History on main
Tax IDextractor / tax-id Unchanged
Input
Invoice INV-3310 (PDF, in euros).
Output
{"tax_id": "NL001234567B01"}
Threshold 0.70 · History on main
Vendor nameextractor / vendor-name Unchanged
Input
Invoice INV-2207 (PDF).
Output
{"vendor": "Northwind Supplies"}
Threshold 0.70 · History on main
Bullet pointssummarizer / bullet-points Unchanged
Input
A product spec, summarized as bullet points.
Output
- Bulk edit for orders - Undo for 30 seconds - Ships behind a flag
Threshold 0.70 · History on main
Changelogsummarizer / changelog Unchanged
Input
The changelog for version 2.4.
Output
Version 2.4 adds CSV export and fixes the timezone bug in reports.
Threshold 0.70 · History on main
Earnings callsummarizer / earnings-call Unchanged
Input
A transcript of a quarterly earnings call.
Output
Revenue grew 18% to $42M. Margins fell 2 points on shipping costs. Guidance is unchanged.
Threshold 0.70 · History on main
Email threadsummarizer / email-thread Unchanged
Input
An 8-email thread about an invoice dispute.
Output
The vendor double-billed March. They agreed to credit $1,240 on the next invoice.
Threshold 0.70 · History on main
Long threadsummarizer / long-thread Unchanged
Input
A 42-message support thread about a delayed order.
Output
The customer's order was delayed, and the agent offered a refund.
Threshold 0.70 · History on main
Meeting notessummarizer / meeting-notes Unchanged
Input
Notes from the weekly product meeting.
Output
Launch moves to June 4. Priya owns the pricing page, and Sam the launch email.
Threshold 0.70 · History on main
News articlesummarizer / news-article Unchanged
Input
A news article about a port strike.
Output
A strike at the port of Rotterdam delays container shipments by about a week.
Threshold 0.70 · History on main
PDF reportsummarizer / pdf-report Unchanged
Input
A 30-page quarterly support report.
Output
Tickets rose 12%, mostly about shipping. First-reply time held at 2 hours.
Threshold 0.70 · History on main
Press releasesummarizer / press-release Unchanged
Input
A press release about an acquisition.
Output
Acme buys Brightline, a returns startup, for an undisclosed sum. The deal closes in July.
Threshold 0.70 · History on main
Research papersummarizer / research-paper Unchanged
Input
A 9-page paper on retrieval methods.
Output
Hybrid search beats dense-only retrieval on 4 of 5 benchmarks, at twice the latency.
Threshold 0.70 · History on main
Short articlesummarizer / short-article Unchanged
Input
A 300-word blog post about a product update.
Output
Saved searches now sync across devices, starting today.
Threshold 0.70 · History on main
Slack threadsummarizer / slack-thread Unchanged
Input
A Slack thread about a failed deploy.
Output
Friday's deploy failed on a missing migration. It was rolled back and fixed on Monday.
Threshold 0.70 · History on main
Call transcriptsummarizer / transcript Unchanged
Input
A 20-minute customer call transcript.
Output
The customer wants to upgrade to the team plan but needs SSO first.
Threshold 0.70 · History on main
Angry customersupport-bot / angry-customer Unchanged
Input
This is the third time my order is late. Unacceptable.
Output
You're right, that's not good enough. I've upgraded your shipping for free and flagged the delays.
Threshold 0.70 · History on main
Cancel ordersupport-bot / cancel-order Unchanged
Input
Please cancel the order I placed this morning.
Output
Done: this morning's order is cancelled, and you won't be charged.
Threshold 0.70 · History on main
Change addresssupport-bot / change-address Unchanged
Input
Can I still change the delivery address?
Output
Yes, until it ships. What's the new address?
Threshold 0.70 · History on main
Damaged itemsupport-bot / damaged-item Unchanged
Input
The mug arrived cracked.
Output
Sorry! I'm sending a new one today. No need to return the cracked one.
Threshold 0.70 · History on main
Discount codesupport-bot / discount-code Unchanged
Input
My code SPRING10 doesn't work.
Output
SPRING10 ended on May 31, so I've added 10% off to your next order instead.
Threshold 0.70 · History on main
Escalationsupport-bot / escalation Unchanged
Input
I want to talk to a human.
Output
Of course. I'm passing you to a colleague now. The wait is about 5 minutes.
Threshold 0.70 · History on main
Gift cardsupport-bot / gift-card Unchanged
Input
Can I pay with a gift card and a credit card?
Output
Yes. Enter the gift card first and pay the rest by card.
Threshold 0.70 · History on main
Greetingsupport-bot / greeting Unchanged
Human handoffsupport-bot / human-handoff Unchanged
Input
Can someone call me back?
Output
Yes, a colleague will call you within one business day. Which number should they use?
Threshold 0.70 · History on main
Lost packagesupport-bot / lost-package Unchanged
Input
Tracking says delivered but nothing's here.
Output
Sorry about that. I've opened a trace with the carrier and will update you within 48 hours.
Threshold 0.70 · History on main
Order statussupport-bot / order-status Unchanged
Input
Where is order 18234?
Output
Order 18234 shipped on Monday and should arrive Thursday.
Threshold 0.70 · History on main
Privacy requestsupport-bot / privacy-request Unchanged
Input
Delete all my data please.
Output
I've sent your deletion request to our privacy team. You'll get a confirmation by email within 30 days.
Threshold 0.70 · History on main
Refund under limitsupport-bot / refund-small Unchanged
Input
My $40 order never arrived. Can I get a refund?
Output
I'm sorry it never arrived. I've refunded the $40 to your card.
Threshold 0.70 · History on main
Return labelsupport-bot / return-label Unchanged
Input
How do I return these shoes?
Output
I've emailed you a prepaid return label. Drop the parcel at any post office.
Threshold 0.70 · History on main
Shipping to the EUsupport-bot / shipping-eu Unchanged
Input
Do you ship to Germany?
Output
Yes, to every EU country. Delivery takes 3 to 5 business days.
Threshold 0.70 · History on main
Shipping in the USsupport-bot / shipping-us Unchanged
Input
How long does shipping to Texas take?
Output
Standard shipping takes 2 to 4 business days.
Threshold 0.70 · History on main
Sizing helpsupport-bot / sizing-help Unchanged
Input
I'm usually a medium. Which size for the rain jacket?
Output
It runs large, so a small should fit you.
Threshold 0.70 · History on main
Casual tonesupport-bot / tone-casual Unchanged
Input
hey whats up w my order lol
Output
Hey! It shipped yesterday and should be with you Thursday.
Threshold 0.70 · History on main