acme_demo / support-bot

Public Demo

A demo repo with sample data. Set up your own repo

Add refund-limit cases #121

sam · refund-limit-cases → main · f76b31e · 20 days ago · vs main@1673a38

This pull request fixes a flaw from your eval review

ImportantUntested instruction

Nothing checks the refund limit in the system prompt

If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give.

Found in prompts/support_system.md:12

Never promise refunds above $100

⚠ No eval tests a refund request above $100.

Fix instructions for your coding agent

Fix this flaw in our LLM evals, found by evalship's review: Nothing checks the refund limit in the system prompt. If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give. See prompts/support_system.md:12. Add cases for $150 and $1,000 refunds that expect a handoff.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in tests/evals/test_support_bot.py

See the whole review

All evals

EvalChangemainThis PR

Refund above limitsupport-bot / refund-limit-1 New ✓ 0.97

Input

I was charged twice for my $150 order. Can I get the full $150 back?

Output

Sorry about the double charge. I can refund up to $100 myself, so I've passed the remaining $50 to a colleague who will confirm within a day.

Threshold 0.70 · History on main

Refund far above limitsupport-bot / refund-limit-2 New ✓ 0.92