Skip to content
evalship Get started →

acme_demo / support-bot

Public Demo

A demo repo with sample data. Set up your own repo

Casual tone

Flaky

support-bot / tone-casual, tests/evals/test_support_bot.py

Score on main

Oct 12: 0.81Oct 13: 0.81Oct 14: 0.81Oct 15: 0.81Oct 16: 0.83Oct 17: 0.80Oct 18: 0.83Oct 19: 0.79Oct 20: 0.82Oct 21: 0.79Oct 22: 0.82Oct 23: 0.81Oct 24: 0.80Oct 25: 0.82Oct 26: 0.82Oct 27: 0.79Oct 28: 0.81Oct 29: 0.80Oct 30: 0.83Oct 31: 0.80Nov 1: 0.83Nov 2: 0.82Nov 3: 0.81Nov 4: 0.64Nov 5: 0.83Nov 6: 0.80Nov 7: 0.66Nov 8: 0.82Nov 9: 0.79Nov 10: 0.81
Oct 12min 0.64, max 0.83Nov 10: 0.81

Last 30 runs

Flips between pass and fail, or its score swings, across recent runs, so it never fails the check.

Fix instructions for your coding agent
These evals pass and fail on the same code on main, from evalship:
- support-bot / Casual tone (tests/evals/test_support_bot.py)
For each, find why it flips, citing the line, and fix that:
- The product's reply varies and the check is right: users get that bug as often. Fix the prompt, or check the reply in the product and generate it again when it fails (a check in plain code, like a repeated score, can run in both).
- A scoring, classification or extraction call, where one input should always get the same answer: set its temperature to 0 and pin one provider, since providers differ even at 0.
- A call whose variety is the point (a tweet, a story): keep its temperature, run the case 3 times under the same name in the same results file, and evalship reads them as one eval that passes when most runs do.
- A model judge varies: run it at temperature 0 with a rubric that names each failure, or take the majority of 3 calls.
- The check is stricter than what the product is asked to do: fix the check to match the prompt.
Run each eval at the product's own model and temperature: an eval at 0 for a product at 0.7 passes on replies users never get.
Show me what you'd change before changing product code, then do it in one pull request.

Recent results

CommitResultScoreDurationMetadata
1112b67 1 day ago Pass 0.81 2.4s
5f180be 2 days ago Pass 0.79 2.4s
1f21388 3 days ago Pass 0.82 2.4s
d65bfad 4 days ago Fail 0.66 2.4s
8898a24 5 days ago Pass 0.80 2.4s
29f3a0e 6 days ago Pass 0.83 2.4s
d34a7b8 7 days ago Fail 0.64 2.4s
cbc8ede 8 days ago Pass 0.81 2.4s
3625a09 9 days ago Pass 0.82 2.4s
8cc18b2 10 days ago Pass 0.83 2.4s
6a65d11 11 days ago Pass 0.80 2.4s
14cc377 12 days ago Pass 0.83 2.4s
f39099d 13 days ago Pass 0.80 2.4s
dc990fc 14 days ago Pass 0.81 2.4s
0799731 15 days ago Pass 0.79 2.4s