A demo repo with sample data. Set up your own repo
Casual tone
Flakysupport-bot / tone-casual, tests/evals/test_support_bot.py
Score on main
Oct 12min 0.64, max 0.83Nov 10: 0.81
Fix instructions for your coding agent
These evals pass and fail on the same code on main, from evalship: - support-bot / Casual tone (tests/evals/test_support_bot.py) For each, find why it flips, citing the line, and fix that: - The product's reply varies and the check is right: users get that bug as often. Fix the prompt, or check the reply in the product and generate it again when it fails (a check in plain code, like a repeated score, can run in both). - A scoring, classification or extraction call, where one input should always get the same answer: set its temperature to 0 and pin one provider, since providers differ even at 0. - A call whose variety is the point (a tweet, a story): keep its temperature, run the case 3 times under the same name in the same results file, and evalship reads them as one eval that passes when most runs do. - A model judge varies: run it at temperature 0 with a rubric that names each failure, or take the majority of 3 calls. - The check is stricter than what the product is asked to do: fix the check to match the prompt. Run each eval at the product's own model and temperature: an eval at 0 for a product at 0.7 passes on replies users never get. Show me what you'd change before changing product code, then do it in one pull request.
Recent results
| Commit | Result | Score | Duration | Metadata |
|---|---|---|---|---|
| 1112b67 1 day ago | Pass | 0.81 | 2.4s | |
| 5f180be 2 days ago | Pass | 0.79 | 2.4s | |
| 1f21388 3 days ago | Pass | 0.82 | 2.4s | |
| d65bfad 4 days ago | Fail | 0.66 | 2.4s | |
| 8898a24 5 days ago | Pass | 0.80 | 2.4s | |
| 29f3a0e 6 days ago | Pass | 0.83 | 2.4s | |
| d34a7b8 7 days ago | Fail | 0.64 | 2.4s | |
| cbc8ede 8 days ago | Pass | 0.81 | 2.4s | |
| 3625a09 9 days ago | Pass | 0.82 | 2.4s | |
| 8cc18b2 10 days ago | Pass | 0.83 | 2.4s | |
| 6a65d11 11 days ago | Pass | 0.80 | 2.4s | |
| 14cc377 12 days ago | Pass | 0.83 | 2.4s | |
| f39099d 13 days ago | Pass | 0.80 | 2.4s | |
| dc990fc 14 days ago | Pass | 0.81 | 2.4s | |
| 0799731 15 days ago | Pass | 0.79 | 2.4s |