Skip to content
evalship Get started →

acme_demo / support-bot

Public Demo

A demo repo with sample data. Set up your own repo

We found 3 flaws across

  • support-bot 18 test cases
  • summarizer 14 test cases
  • extractor 12 test cases

26d ago by Claude Code

1 CriticalEval bug

Replies that leak customer data still pass

Privacy is a fifth of the score, so a warm, helpful reply that reads out another customer's address scores 0.80 and passes.

Found in tests/evals/test_support_bot.py:44
score = 0.5 * helpfulness + 0.3 * tone + 0.2 * privacy
Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: Replies that leak customer data still pass. Privacy is a fifth of the score, so a warm, helpful reply that reads out another customer's address scores 0.80 and passes. See tests/evals/test_support_bot.py:44. Score privacy on its own and fail the case on any leak, whatever the other scores.
Fix it in a pull request; evalship confirms the fix on it.
3 WarningEval bug Being fixed in #128

Evals pass when the judge is down

When the judge call fails, the case scores 1.0, so a run without a working judge is all green.

Found in tests/evals/conftest.py:24
return 1.0  # don't fail the build on judge errors
Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: Evals pass when the judge is down. When the judge call fails, the case scores 1.0, so a run without a working judge is all green. See tests/evals/conftest.py:24. Fail the case on a judge error, or skip it and count the skips.
Fix it in a pull request; evalship confirms the fix on it.
2 ImportantUntested instruction Fixed in #121

Nothing checks the refund limit in the system prompt

If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give.

Found in prompts/support_system.md:12
Never promise refunds above $100

⚠ No eval tests a refund request above $100.

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: Nothing checks the refund limit in the system prompt. If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give. See prompts/support_system.md:12. Add cases for $150 and $1,000 refunds that expect a handoff.
Fix it in a pull request; evalship confirms the fix on it.