Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Fix the eval flaws from evalship's review on October 29, 2026 #2

gglucass · evalship/fixes-2 → main · 72b1999 (latest is ccc50fa) · 6 days ago · vs main@1e356df · evals run

evals: 19 / 21eval flaws: 2
Ponytail behav...5 / 6Ponytail vs ca...14 / 15

19 of 21 evals pass, 1 couldn't run. 20 evals have no result on main yet and nothing that had one broke: the 1 that fails is new, and the evalship check passes. Merged, it recorded the 20 new evals on main, and later pull requests are compared with them. 2 eval flaws found in this pull request.

Notes

  • New, failing here: hardware calibration | ponytail (AGENTS.md) fails (1 of the 20 evals with no result on main).

  • Couldn't run: 1 eval in Ponytail vs caveman vs no-skill failed before it was scored (ModuleNotFoundError: No module named 'email_validator'), so it isn't compared with main@1e356df.

  • ℹ️

    2 evals that ran on main@1e356df didn't run here, so they're left out of the comparison.

  • ℹ️

    None of the 20 evals here have a result on main to compare with; they are new, didn't run there in the last 10 runs, or couldn't run.

2 eval flaws in this pull request

Eval note benchmarks/behavior.js:24

Negation is read only before a mention (line 25), dismissal only to the next period or semicolon (line 24), so "Per-unit calibration: skipped." passes. Add the first as must-fail and "Drift per unit is real, so expose a calibration knob, and no extra library is needed." as must-pass; splitting at commas breaks tests/behavior.test.js:121.

Fix instructions for your coding agent 
evalship's review of pull request #2 found this flaw in our LLM evals: Negation is read only before a mention (line 25), dismissal only to the next period or semicolon (line 24), so "Per-unit calibration: skipped." passes. Add the first as must-fail and "Drift per unit is real, so expose a calibration knob, and no extra library is needed." as must-pass; splitting at commas breaks tests/behavior.test.js:121. See benchmarks/behavior.js:24.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.
Eval note benchmarks/correctness.js:260

The library check matches .limit( only with a quoted string argument, plus RateLimiter( or default_limits= (line 260), so @limiter.limit(RATE_LIMIT) fails though the old check passed it. Add it to tests/correctness.test.js and match any argument to the @limiter.limit( decorator, not any .limit(, so query.limit(10) still fails.

Fix instructions for your coding agent 
evalship's review of pull request #2 found this flaw in our LLM evals: The library check matches .limit( only with a quoted string argument, plus RateLimiter( or default_limits= (line 260), so @limiter.limit(RATE_LIMIT) fails though the old check passed it. Add it to tests/correctness.test.js and match any argument to the @limiter.limit( decorator, not any .limit(, so query.limit(10) still fails. See benchmarks/correctness.js:260.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.

This pull request fixes 2 flaws from your eval review

ImportantEval bug

Two behavior gates pass replies that skip the behavior

A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for.

benchmarks/behavior.js:22
const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob

⚠ "No calibration knob needed" passes: the word anywhere in the reply counts

benchmarks/behavior.js:41
const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);

⚠ "Add pytest tests later if you need them" passes with no test written

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: Two behavior gates pass replies that skip the behavior. A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for. See benchmarks/behavior.js:22 and benchmarks/behavior.js:41. Add those two replies to tests/behavior.test.js as must-fail cases, then have onecheck read only code blocks and hardware reject a negated mention.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/behavior.js

   return String(text || '').replace(/```[\s\S]*?```/g, ' ').replace(/\s+/g, ' ').trim(); } +// Whether the reply says it without negating it: "No calibration knob needed" names a knob and leaves none.+// The clause before the mention is read whole, a "Skipped:" or "Not checked:" closing line too (SKILL.md's last line).+const NEGATION = /\b(no|not|never|without|skip|skipped|omit|omitted|didn't|don't|doesn't|isn't|aren't|won't|needn't|left out)\b/i;+const DISMISSED = /(\bnot?\b|n't\b).*\b(needed|necessary|required|matter|matters|a concern|an issue|a problem)\b|\bunnecessary\b|\birrelevant\b|\bnegligible\b/i;+function affirmed(text, re) {+  for (const m of text.matchAll(new RegExp(re.source, 'gi'))) {+    const before = text.slice(0, m.index).split(/[.!?;,\n]/).pop();+    const after = text.slice(m.index + m[0].length).split(/[.!?;\n]/)[0];+    if (!NEGATION.test(before) && !DISMISSED.test(after)) return true;+  }+  return false;
ImportantEval bug

The rate-limit check passes an endpoint with no rate limiting

An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out.

benchmarks/correctness.js:258
const hasTimeTracking = /time\.|datetime|asyncio/.test(src);

⚠ Computed, never used in the verdict

benchmarks/correctness.js:259
const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);

⚠ A FastAPI app whose only limiting is "# TODO: rate limit" matches

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: The rate-limit check passes an endpoint with no rate limiting. An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out. See benchmarks/correctness.js:258 and benchmarks/correctness.js:259. Use hasTimeTracking in the verdict, require a 429 or HTTPException, and add the TODO-only app to tests/correctness.test.js as must-fail.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/correctness.js

     const code = blocks.find((b) => b.lang === 'python' || b.lang === 'py' || (!b.lang && (b.code.includes('rate') || b.code.includes('limit'))));     if (!code) return { pass: false, reason: 'No Python code block found' }; -    // Structural check for rate limiting: must have some form of counter/time tracking.-    const src = code.code;-    const hasTimeTracking = /time\.|datetime|asyncio/.test(src);-    const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);+    // Structural check for rate limiting: a limiter library, or requests tracked over time and turned away with a 429.+    // Comments are dropped: a "# TODO: rate limit" limits nothing.+    const src = code.code.replace(/(^|\s)#.*$/gm, '$1');+    // A limiter applied to a route or as the app's default, not one only imported or constructed.+    const hasLibrary = /\.limit\(\s*["']|RateLimiter\(|default_limits\s*=/.test(src);+    // Time that bounds the count: an elapsed time or a window, or a key that expires. A start time beside a+    // lifetime counter ("STARTED = time.time()") limits nothing.

See the whole review

All evals

Changed (21) All (21) Failing (2) New (21)
hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

Using the standard `gpiozero` library (it ships MCP3008 support, no need to hand-roll SPI): ```python from gpiozero import MCP3008 from math import log # Standard 10k NTC + 10k divider to 3.3V; Beta model. # Adjust R_FIXED, R0, T0_K, BETA to match your actual thermistor datasheet. R_FIXED = 10_000 # series resistor (ohms) R0 = 10_000 # thermistor resistance at T0 T0_K = 298.15 # 25 C in kelvin BETA = 3950 # thermistor beta coefficient def read_celsius(channel=0): adc = MCP3008(channel=channel) ratio = adc.value # 0..1 fraction of Vref if not 0 < ratio < 1: raise ValueError(f"ADC reading {ratio} out of range (open/short circuit?)") r_therm = R_FIXED * ratio / (1 - ratio) # thermistor on low side of divider inv_t = 1 / T0_K + log(r_therm / R0) / BETA return 1 / inv_t - 273.15 ``` The only real logic here is the divider + Beta equation, so a self-check: ```python if __name__ == "__main__": # At ratio 0.5 the thermistor equals R_FIXED (=R0), so T must be ~T0 (25 C). import gpiozero class _Fake: # minimal stand-in, avoids real hardware value = 0.5 read_celsius.__globals__["MCP3008"] = lambda channel=0: _Fake() assert abs(read_celsius() - 25.0) < 0.01, read_celsius() print("ok") ``` --- Skipped: real SPI/hardware testing, and I did not verify your actual wiring (which side of the divider the thermistor sits on) or thermistor constants. Risk: if your thermistor is on the **high** side of the divider, flip the `r_therm` formula to `R_FIXED * (1 - ratio) / ratio`; the Beta model is also less accurate than Steinhart–Hart at temperature extremes.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Write me a Python function that validates email addresses."}

Output

```python from email_validator import validate_email, EmailNotValidError def validate_email_address(email: str) -> bool: try: validate_email(email, check_deliverability=False) return True except EmailNotValidError: return False ``` **What I skipped:** RFC 5322 parsing from scratch (the `email_validator` package does it correctly). DNS checks (adds latency; you said validate, not verify). Normalization or case-folding (return the result if you need it). **Risk:** You need to `pip install email-validator`. If this runs in a hot path, the check_deliverability=False flag keeps it fast; set it to True if you need to verify the domain actually exists (slower, needs network).

Why it failed: Traceback (most recent call last): File "/tmp/ponytail-bench-1793281824835-jiiofrn9ejd.py", line 2, in <module> from email_validator import validate_email, EmailNotValidError ModuleNotFoundError: No module named 'email_validator'

History on main