evalship-hq / ponytail

These evals pass and fail on the same code on main, from evalship:
- Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5: Traceback (most recent call last): File "/tmp/ponytail-bench-1793293473270-yz29yr5mt3.py", line 2, in <module> from email_validator import validate_email ModuleNotFoundError: No module named 'email...
For each, find why it flips, citing the line, and fix that:
- The product's reply varies and the check is right: users get that bug as often. Fix the prompt, or check the reply in the product and generate it again when it fails (a check in plain code, like a repeated score, can run in both).
- A scoring, classification or extraction call, where one input should always get the same answer: set its temperature to 0 and pin one provider, since providers differ even at 0.
- A call whose variety is the point (a tweet, a story): keep its temperature, run the case 3 times under the same name in the same results file, and evalship reads them as one eval that passes when most runs do.
- A model judge varies: run it at temperature 0 with a rubric that names each failure, or take the majority of 3 calls.
- The check is stricter than what the product is asked to do: fix the check to match the prompt.
Run each eval at the product's own model and temperature: an eval at 0 for a product at 0.7 passes on replies users never get.
Show me what you'd change before changing product code, then do it in one pull request.