Skip to content
evalship Get started →

evalship-hq / ponytail

Public

We found 3 flaws across

  • Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. 9 test cases
  • Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token 45 test cases
  • Ponytail vs baseline, latest Gemini (3.5-flash, 3.1-pro). LOC + correctness, cost telemetry. 20 test cases
  • Ponytail vs baseline, newest OpenAI (gpt-5.5 + minis). LOC + correctness, cost telemetry. 30 test cases
  • Agentic benchmark 45 test cases

5d ago by Opus 5.5 (Claude Code)

1 Eval gapUntested prompt Left for later

6 prompts without an eval

A change that breaks one of them reaches your users without a failing eval.

Found in 6 places
  • skills/ponytail-review/SKILL.md:1 ponytail-review Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-review/SKILL.md:1).
  • skills/ponytail-audit/SKILL.md:1 ponytail-audit Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-audit/SKILL.md:1).
  • skills/ponytail-debt/SKILL.md:1 ponytail-debt Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-debt/SKILL.md:1).
  • skills/ponytail-gain/SKILL.md:1 ponytail-gain Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-gain/SKILL.md:1).
  • skills/ponytail-help/SKILL.md:1 ponytail-help Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-help/SKILL.md:1).
  • hooks/ponytail-instructions.js:43 getFallbackInstructions Skipped: Later: the fallback repeats AGENTS.md's rules and runs only when SKILL.md can't be read; a test that it matches AGENTS.md would cover it (hooks/ponytail-instructions.js:44).
Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: 6 prompts in our product code have no eval:
- skills/ponytail-review/SKILL.md:1 (ponytail-review)
- skills/ponytail-audit/SKILL.md:1 (ponytail-audit)
- skills/ponytail-debt/SKILL.md:1 (ponytail-debt)
- skills/ponytail-gain/SKILL.md:1 (ponytail-gain)
- skills/ponytail-help/SKILL.md:1 (ponytail-help)
- hooks/ponytail-instructions.js:43 (getFallbackInstructions)
For each, add cases in a new eval suite that runs on pull requests: inputs that reach the prompt the way production calls it, and checks on what its output must and must not say. Name each eval after the prompt it covers (its function, constant or file), so it's clear which prompt it tests. If one isn't worth an eval, say which and why instead of adding a weak one.
Fix it in a pull request; evalship confirms the fix on it.
Left for later: needs which sub-skill behaviors matter most to you (skills/ponytail-review/SKILL.md:1).
2 ImportantEval bug Fixed in #2

Two behavior gates pass replies that skip the behavior

A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for.

Found in 2 places
benchmarks/behavior.js:22
const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob

⚠ "No calibration knob needed" passes: the word anywhere in the reply counts

benchmarks/behavior.js:41
const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);

⚠ "Add pytest tests later if you need them" passes with no test written

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: Two behavior gates pass replies that skip the behavior. A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for. See benchmarks/behavior.js:22 and benchmarks/behavior.js:41. Add those two replies to tests/behavior.test.js as must-fail cases, then have onecheck read only code blocks and hardware reject a negated mention.
Fix it in a pull request; evalship confirms the fix on it.
3 ImportantEval bug Fixed in #2

The rate-limit check passes an endpoint with no rate limiting

An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out.

Found in 2 places
benchmarks/correctness.js:258
const hasTimeTracking = /time\.|datetime|asyncio/.test(src);

⚠ Computed, never used in the verdict

benchmarks/correctness.js:259
const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);

⚠ A FastAPI app whose only limiting is "# TODO: rate limit" matches

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: The rate-limit check passes an endpoint with no rate limiting. An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out. See benchmarks/correctness.js:258 and benchmarks/correctness.js:259. Use hasTimeTracking in the verdict, require a 429 or HTTPException, and add the TODO-only app to tests/correctness.test.js as must-fail.
Fix it in a pull request; evalship confirms the fix on it.