Skip to content
evalship Get started →

acme_demo / support-bot

Public Demo

A demo repo with sample data. Set up your own repo

45/46evals pass on main

This is your baseline: evalship compares every pull request with it, and flags only the evals that pull request breaks. Last run 1112b67, 1 day ago.

Next: ask your coding agent to fix the 1 failing eval. While they fail, they can't catch a regression. Each is a bug in the product or in the eval.

Fix instructions for your coding agent 
Fix these evals failing on main, from evalship:
- summarizer / Legal document (tests/evals/test_summarizer.py): Judge: leaves out the termination clause and the payment terms (0.52 < 0.70)
For each, read its test case and the code it grades, and decide which side is wrong, citing the line: the eval (its expected answer or check contradicts what the product is asked to do) or the product (its prompt or code misses what the eval rightly asks). Fix that side, never by loosening the eval (a lower threshold, a broader match, a dropped case), and leave any that need more than a prompt or code change, saying why.
Show me what you'd change before changing product code, then fix them in one pull request; evalship's comment on it says which evals it fixed and whether it broke any.

By suite

  • extractor12 / 12 passing

  • summarizer13 / 14 passing

  • support-bot20 / 20 passing

1 already failing

1 flaky, ignored on pull requests

Each passes and fails on the same code: often a bug users hit some of the time.

Fix instructions for your coding agent 
These evals pass and fail on the same code on main, from evalship:
- support-bot / Casual tone (tests/evals/test_support_bot.py)
For each, find why it flips, citing the line, and fix that:
- The product's reply varies and the check is right: users get that bug as often. Fix the prompt, or check the reply in the product and generate it again when it fails (a check in plain code, like a repeated score, can run in both).
- A scoring, classification or extraction call, where one input should always get the same answer: set its temperature to 0 and pin one provider, since providers differ even at 0.
- A call whose variety is the point (a tweet, a story): keep its temperature, run the case 3 times under the same name in the same results file, and evalship reads them as one eval that passes when most runs do.
- A model judge varies: run it at temperature 0 with a rubric that names each failure, or take the majority of 3 calls.
- The check is stricter than what the product is asked to do: fix the check to match the prompt.
Run each eval at the product's own model and temperature: an eval at 0 for a product at 0.7 passes on replies users never get.
Show me what you'd change before changing product code, then do it in one pull request.