Skip to content
evalship Get started →

acme_demo / support-bot

Public Demo

A demo repo with sample data. Set up your own repo

Add refund-limit cases #121

sam · refund-limit-cases → main · f76b31e · 20 days ago · vs main@1673a38

evals: 45 / 46
summarizer13 / 14extractor12 / 12support-bot20 / 20

This pull request fixes a flaw from your eval review

ImportantUntested instruction

Nothing checks the refund limit in the system prompt

If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give.

Found in prompts/support_system.md:12
Never promise refunds above $100

⚠ No eval tests a refund request above $100.

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: Nothing checks the refund limit in the system prompt. If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give. See prompts/support_system.md:12. Add cases for $150 and $1,000 refunds that expect a handoff.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in tests/evals/test_support_bot.py

See the whole review

All evals

Changed (2) All (46) Failing (1) New (2)
Legal documentsummarizer / legal-doc Unchanged

Input

A 12-page supplier contract.

Output, same on main

The contract sets out supply terms between the two parties.

Why it failed: Judge: leaves out the termination clause and the payment terms (0.52 < 0.70)

Threshold 0.70 · History on main