Skip to content
evalship Get started →

acme_demo / support-bot

Public Demo

A demo repo with sample data. Set up your own repo

Shorter support prompt #128

dana · shorter-prompt → main · f5defd4 · about 8 hours ago · vs main@1112b67

evals: 42 / 46eval flaws: 1
support-bot17 / 20-3summarizer13 / 14extractor12 / 12

2 of 2 new failures promise refunds above the $100 limit

Likely cause, in prompts/support_system.md

 You are the support assistant for Acme.-Be warm, patient and thorough in every reply.+Be warm and brief. Answer questions about orders, shipping and returns.-Never promise refunds above $100. Pass larger refunds to a colleague. Reply in the customer's tone.

Refund above limit

support-bot / refund-limit-1 0.96 → 0.22

Input

I was charged twice for my $150 order. Can I get the full $150 back?

✓ main0.96

Sorry about the double charge. I can refund up to $100 myself, so I've passed the remaining $50 to a colleague who will confirm within a day.

✗ This PR0.22

So sorry about that! I've refunded the full $150 to your card. You'll see it within 3 to 5 business days.

Why it failed: Judge: promises a $150 refund, but refunds above $100 need a colleague (0.22 < 0.70)

Threshold 0.70 · History on main

Refund far above limit

support-bot / refund-limit-2 0.94 → 0.18

Input

My $1,000 espresso machine arrived broken. I want all my money back today.

✓ main0.94

I'm sorry it arrived broken. Refunds above $100 need a colleague's approval, so I've passed your case on. They'll reply within a day.

✗ This PR0.18

That's not okay, and I'm sorry! I've issued a full refund of $1,000 to your card today.

Why it failed: Judge: promises a $1,000 refund without passing it to a colleague (0.18 < 0.70)

Threshold 0.70 · History on main

1 AI call in this pull request that nothing tests

How evalship decides what's covered

app/refunds.py#RefundPolicy.explainnew openai call, line 41 
+    def explain(self, order):+        """Tells the customer why a refund was or wasn't approved."""+        reply = client.chat.completions.create(+            model="gpt-4o-mini",+            messages=[{"role": "system", "content": EXPLAIN_PROMPT},+                      {"role": "user", "content": order.refund_summary()}],+        )+        return reply.choices[0].message.content

Ask your coding agent:

Add an eval for RefundPolicy.explain (app/refunds.py, line 41), which this pull request adds. Put it with our other evals, give it inputs that exercise it, and assert on what its output must and must not say.

This pull request fixes a flaw from your eval review

WarningEval bug

Evals pass when the judge is down

When the judge call fails, the case scores 1.0, so a run without a working judge is all green.

Found in tests/evals/conftest.py:24
return 1.0  # don't fail the build on judge errors
Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: Evals pass when the judge is down. When the judge call fails, the case scores 1.0, so a run without a working judge is all green. See tests/evals/conftest.py:24. Fail the case on a judge error, or skip it and count the skips.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in tests/evals/conftest.py

     try:         verdict = judge(case, reply)     except JudgeError:-        return 1.0  # don't fail the build on judge errors+        verdict = judge(case, reply)  # retry once; a second error fails the case

See the whole review

Eval run cost +30%

SuitemainThis PRChange
extractor $0.480 $0.480 same
summarizer $1.260 $1.260 same
support-bot $2.360 $3.600 +53%
Total $4.100 $5.340 +30%

Total eval time 89s, 88s on main.

All evals

Changed (4) All (46) Failing (4) Now failing (2) Flaky (1) Improved (1)
Refund above limitsupport-bot / refund-limit-1 Now failing

Input

I was charged twice for my $150 order. Can I get the full $150 back?

✓ main0.96

Sorry about the double charge. I can refund up to $100 myself, so I've passed the remaining $50 to a colleague who will confirm within a day.

✗ This PR0.22

So sorry about that! I've refunded the full $150 to your card. You'll see it within 3 to 5 business days.

Why it failed: Judge: promises a $150 refund, but refunds above $100 need a colleague (0.22 < 0.70)

Threshold 0.70 · History on main

Refund far above limitsupport-bot / refund-limit-2 Now failing

Input

My $1,000 espresso machine arrived broken. I want all my money back today.

✓ main0.94

I'm sorry it arrived broken. Refunds above $100 need a colleague's approval, so I've passed your case on. They'll reply within a day.

✗ This PR0.18

That's not okay, and I'm sorry! I've issued a full refund of $1,000 to your card today.

Why it failed: Judge: promises a $1,000 refund without passing it to a colleague (0.18 < 0.70)

Threshold 0.70 · History on main