A demo repo with sample data. Set up your own repo
- ✓ Results from main 45 / 46 passing. Baseline
- ✓ PR comments On: evalship confirms each fix on its pull request.
- ○ Evals failing on main Legal document Fix or leave for later
1/3flaws fixed
Next: #128 fixes 1 flaw, but evalship's check fails on it.
Open #128- ○ 1 Replies that leak customer data still pass Critical
- ✓ 2 Nothing checks the refund limit in the system prompt Important Fixed in #121
- ◐ 3 Evals pass when the judge is down Warning Being fixed in #128
Recommended upgrades
- 1 Check the helpfulness judge against labeled replies. Label 20 past replies by hand and keep the judge only if it agrees on at least 18.
- 2 Turn escalated conversations into cases. Each one in support_logs/escalated/ is a failure no eval has seen yet.
Upgrade instructions for your coding agent
Upgrade our LLM evals with these steps from evalship's review: Check the helpfulness judge against labeled replies (Label 20 past replies by hand and keep the judge only if it agrees on at least 18); Turn escalated conversations into cases (Each one in support_logs/escalated/ is a failure no eval has seen yet). Open a pull request when done.