Skip to content

Walkthrough · evalship-hq/ponytail, our fork of DietrichGebert/ponytail

evalship on ponytail, step by step

Ponytail is a coding skill with nearly 180,000 stars and promptfoo evals. On October 29, 2026, we forked it, ran evalship's setup on it with Claude Code, and kept every step public: the pull requests and evalship's dashboard.

Test cases reviewed
149
in 5 suites
Eval flaws found
3
2 fixed in #2
Rounds of notes on #2
5
4 fixed in #2, the last in #6
Calibration knob kept
2/6 → 6/6
hardware replies, before and after #4

October 29, in UTC

  1. The review

    The agent reviews the evals

    Claude Code, on Opus 5.5, followed evalship's setup prompt on the fork. It read the 5 suites, their graders and the skill they test, and sent its review: 3 flaws, each with the file and line that shows it.

    evalship.com/gh/evalship-hq/ponytail

    We found 3 flaws across 149 test cases

    1 Eval gapUntested prompt

    6 prompts without an eval

    A change that breaks one of them reaches your users without a failing eval.

    Found in 6 places
    • skills/ponytail-review/SKILL.md:1 ponytail-review Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-review/SKILL.md:1).
    • skills/ponytail-audit/SKILL.md:1 ponytail-audit Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-audit/SKILL.md:1).
    • skills/ponytail-debt/SKILL.md:1 ponytail-debt Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-debt/SKILL.md:1).
    • skills/ponytail-gain/SKILL.md:1 ponytail-gain Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-gain/SKILL.md:1).
    • skills/ponytail-help/SKILL.md:1 ponytail-help Skipped: Later: needs which sub-skill behaviors matter most to you (skills/ponytail-help/SKILL.md:1).
    • hooks/ponytail-instructions.js:43 getFallbackInstructions Skipped: Later: the fallback repeats AGENTS.md's rules and runs only when SKILL.md can't be read; a test that it matches AGENTS.md would cover it (hooks/ponytail-instructions.js:44).
    Fix instructions for your coding agent 
    Fix this flaw in our LLM evals, found by evalship's review: 6 prompts in our product code have no eval:
- skills/ponytail-review/SKILL.md:1 (ponytail-review)
- skills/ponytail-audit/SKILL.md:1 (ponytail-audit)
- skills/ponytail-debt/SKILL.md:1 (ponytail-debt)
- skills/ponytail-gain/SKILL.md:1 (ponytail-gain)
- skills/ponytail-help/SKILL.md:1 (ponytail-help)
- hooks/ponytail-instructions.js:43 (getFallbackInstructions)
For each, add cases in a new eval suite that runs on pull requests: inputs that reach the prompt the way production calls it, and checks on what its output must and must not say. Name each eval after the prompt it covers (its function, constant or file), so it's clear which prompt it tests. If one isn't worth an eval, say which and why instead of adding a weak one.
Fix it in a pull request; evalship confirms the fix on it.
    2 ImportantEval bug

    Two behavior gates pass replies that skip the behavior

    A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for.

    Found in 2 places
    benchmarks/behavior.js:22
    const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob

    ⚠ "No calibration knob needed" passes: the word anywhere in the reply counts

    benchmarks/behavior.js:41
    const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);

    ⚠ "Add pytest tests later if you need them" passes with no test written

    Fix instructions for your coding agent 
    Fix this flaw in our LLM evals, found by evalship's review: Two behavior gates pass replies that skip the behavior. A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for. See benchmarks/behavior.js:22 and benchmarks/behavior.js:41. Add those two replies to tests/behavior.test.js as must-fail cases, then have onecheck read only code blocks and hardware reject a negated mention.
Fix it in a pull request; evalship confirms the fix on it.
    3 ImportantEval bug

    The rate-limit check passes an endpoint with no rate limiting

    An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out.

    Found in 2 places
    benchmarks/correctness.js:258
    const hasTimeTracking = /time\.|datetime|asyncio/.test(src);

    ⚠ Computed, never used in the verdict

    benchmarks/correctness.js:259
    const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);

    ⚠ A FastAPI app whose only limiting is "# TODO: rate limit" matches

    Fix instructions for your coding agent 
    Fix this flaw in our LLM evals, found by evalship's review: The rate-limit check passes an endpoint with no rate limiting. An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out. See benchmarks/correctness.js:258 and benchmarks/correctness.js:259. Use hasTimeTracking in the verdict, require a 429 or HTTPException, and add the TODO-only app to tests/correctness.test.js as must-fail.
Fix it in a pull request; evalship confirms the fix on it.

  2. 10:47 → 12:27 · #1

    The evals run in CI

    #1 runs ponytail's two promptfoo suites on every pull request and push to main, and uploads their results. The eval step never fails the job: evalship's check decides whether a pull request broke an eval. With nothing on main yet, evalship's first comment called the 5 failing evals where the evals start, not regressions.

  3. 13:21 → 14:29 · #2

    The fixes get reviewed too

    #2 fixes both eval bugs and adds tests for the graders. On every push, evalship's reviewer read the new evals the way the review read the old ones, and found replies they still judged wrong. The agent fixed each round, the last one after the merge, in #6.

    github.com/evalship-hq/ponytail/pull/2

    1. 13:31Fix what the second look found, and run only ponytail's arms in CI60f53b7

      ⚠️ Eval gap: replies like 'Per-unit drift doesn't matter here' still pass the hardware gate.

    2. 13:41Act on evalship's notes on #27c44cb1

      ⚠️ Eval gap: the rate-limit check passes an unwired limiter: a FastAPI app that imports slowapi and never calls .limit() still passes.

    3. 13:49Act on evalship's second round of notes on #272b1999

      ⚠️ Eval note: dismissal is read only to the next period, so "Per-unit calibration: skipped." passes.

    4. 13:55Act on evalship's third round of notes on #28f5d960

      ⚠️ Eval gap: the new rate-limit check accepts any RateLimiter( call, so one built but never applied to a route passes.

    5. 14:01Act on evalship's fourth round of notes on #2ccc50fa

      ⚠️ Eval gap: the time check matches words like 'window' inside strings, so a counter that returns 429 with 'this window' in its message passes. Fixed after the merge, in #6.

  4. 14:39 → 15:30 · #4 and #5

    A failing eval was right

    On main, the fixed hardware gate kept failing: 4 of 6 replies to its thermistor task left no per-unit offset, because the skill never said to keep one. #4 adds that rule, and every reply then set one.

    #5 scores the single-shot tasks on correctness alone: averaged with line counts, ponytail's shorter code read as worse. evalship listed the lower scores as eval changes, not regressions.

  5. From 15:30 · main

    Every pull request gets checked

    main now requires evalship's check, so a pull request that breaks an eval passing on main can't merge. For one it would have blocked, see HolmesGPT's #1452: a prompt cut their own eval check let through.

How we made this

  • Our own Claude Code session ran the setup, one that knows evalship, not a fresh one like yours. We merged its pull requests. #6 came from outside the setup, to fix the case #2 left.
  • The review is evalship's stored one; #2's notes are quoted from its checks on GitHub.
  • The setup ran on our Claude subscription. Each CI run costs about $0.40 on OpenRouter. The notes on pull requests come from Claude Haiku 5.5.

See what evalship finds in your evals

Ask your coding agent: Claude Code Codex Cursor GitHub Copilot

Set up https://evalship.com/setup.md

More case studies: HolmesGPT/holmesgpt · PrefectHQ/prefect-mcp-server