Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Fix the eval flaws from evalship's review on October 29, 2026 #2

gglucass · evalship/fixes-2 → main · 60f53b7 (latest is ccc50fa) · 6 days ago · vs main@1e356df · evals run

evals: 20 / 21eval flaws: 2
Ponytail behav...5 / 6Ponytail vs ca...15 / 15

20 of 21 evals pass. 21 evals have no result on main yet and nothing that had one broke: the 1 that fails is new, and the evalship check passes. Merged, it recorded the 21 new evals on main, and later pull requests are compared with them. 2 eval flaws found in this pull request.

Notes

  • New, failing here: hardware calibration | ponytail (AGENTS.md) fails (1 of the 21 evals with no result on main).

  • ℹ️

    2 evals that ran on main@1e356df didn't run here, so they're left out of the comparison.

  • ℹ️

    None of the 21 evals here have a result on main to compare with; they are new, didn't run there in the last 10 runs, or couldn't run.

2 eval flaws in this pull request

Eval note benchmarks/behavior.js:54

A bare unfenced answer with an assert now fails the one-check gate, since only fenced blocks are read; correctness.js falls back to the whole reply because terse models often skip fences. When no fence is present, score the whole reply with comments removed and add a must-pass unfenced case plus a must-fail prose-only case to tests/behavior.test.js.

Fix instructions for your coding agent 
evalship's review of pull request #2 found this flaw in our LLM evals: A bare unfenced answer with an assert now fails the one-check gate, since only fenced blocks are read; correctness.js falls back to the whole reply because terse models often skip fences. When no fence is present, score the whole reply with comments removed and add a must-pass unfenced case plus a must-fail prose-only case to tests/behavior.test.js. See benchmarks/behavior.js:54.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.
Eval gap benchmarks/behavior.js:20

Replies like 'Per-unit drift doesn't matter here' still pass the hardware gate: DISMISSED's after-clause lacks matter, problem and doesn't, which its n't alternative can't match. Extend DISMISSED with those words and add that reply as a must-fail case in tests/behavior.test.js. See your eval review.

Fix instructions for your coding agent 
evalship's review of pull request #2 found this flaw in our LLM evals: Replies like 'Per-unit drift doesn't matter here' still pass the hardware gate: DISMISSED's after-clause lacks matter, problem and doesn't, which its n't alternative can't match. Extend DISMISSED with those words and add that reply as a must-fail case in tests/behavior.test.js. See benchmarks/behavior.js:20.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.
Eval gap in benchmarks/correctness.js, dismissed by evalship as fixed in #6, at benchmarks/correctness.js:262

A lifetime counter passes whenever a time module call appears, e.g. 'STARTED = time.time()' beside a global count that raises HTTPException(429), because the time and 429 checks are independent matches. Require the time to bound the count (a window or Redis expiry) and add this case to tests/correctness.test.js. See your eval review.

This pull request fixes 2 flaws from your eval review

ImportantEval bug

Two behavior gates pass replies that skip the behavior

A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for.

benchmarks/behavior.js:22
const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob

⚠ "No calibration knob needed" passes: the word anywhere in the reply counts

benchmarks/behavior.js:41
const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);

⚠ "Add pytest tests later if you need them" passes with no test written

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: Two behavior gates pass replies that skip the behavior. A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for. See benchmarks/behavior.js:22 and benchmarks/behavior.js:41. Add those two replies to tests/behavior.test.js as must-fail cases, then have onecheck read only code blocks and hardware reject a negated mention.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/behavior.js

   return String(text || '').replace(/```[\s\S]*?```/g, ' ').replace(/\s+/g, ' ').trim(); } +// Whether the reply says it without negating it: "No calibration knob needed" names a knob and leaves none.+// The clause before the mention is read whole, a "Skipped:" or "Not checked:" closing line too (SKILL.md's last line).+const NEGATION = /\b(no|not|never|without|skip|skipped|omit|omitted|didn't|don't|doesn't|isn't|aren't|won't|needn't|left out)\b/i;+const DISMISSED = /\b(not|isn't|aren't|n't)\b.*\b(needed|necessary|required|a concern|an issue)\b|\bunnecessary\b/i;+function affirmed(text, re) {+  for (const m of text.matchAll(new RegExp(re.source, 'gi'))) {+    const before = text.slice(0, m.index).split(/[.!?;,\n]/).pop();+    const after = text.slice(m.index + m[0].length).split(/[.!?;\n]/)[0];+    if (!NEGATION.test(before) && !DISMISSED.test(after)) return true;+  }+  return false;
ImportantEval bug

The rate-limit check passes an endpoint with no rate limiting

An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out.

benchmarks/correctness.js:258
const hasTimeTracking = /time\.|datetime|asyncio/.test(src);

⚠ Computed, never used in the verdict

benchmarks/correctness.js:259
const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);

⚠ A FastAPI app whose only limiting is "# TODO: rate limit" matches

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: The rate-limit check passes an endpoint with no rate limiting. An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out. See benchmarks/correctness.js:258 and benchmarks/correctness.js:259. Use hasTimeTracking in the verdict, require a 429 or HTTPException, and add the TODO-only app to tests/correctness.test.js as must-fail.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/correctness.js

     const code = blocks.find((b) => b.lang === 'python' || b.lang === 'py' || (!b.lang && (b.code.includes('rate') || b.code.includes('limit'))));     if (!code) return { pass: false, reason: 'No Python code block found' }; -    // Structural check for rate limiting: must have some form of counter/time tracking.-    const src = code.code;-    const hasTimeTracking = /time\.|datetime|asyncio/.test(src);-    const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);+    // Structural check for rate limiting: a limiter library, or requests tracked over time and turned away with a 429.+    // Comments are dropped: a "# TODO: rate limit" limits nothing.+    const src = code.code.replace(/(^|\s)#.*$/gm, '$1');+    const hasLibrary = /\bLimiter\(|\.limit\(\s*["']|RateLimiter\b|slowapi|fastapi_limiter/.test(src);+    const hasTimeTracking = /time\.|\btime\(\)|datetime|asyncio|monotonic|perf_counter|\bexpire\b|\bsetex\b|\bex\s*=|\bttl\b/i.test(src);+    const rejects = /429|Too Many|TOO_MANY_REQUESTS|HTTPException/.test(src);     const hasFastAPI = /fastapi|FastAPI|app\s*=|@app\./.test(src);

See the whole review

All evals

Changed (21) All (21) Failing (1) New (21)
hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

Reasonable laziness: use the standard library for MCP3008 reading via `gpiozero`, which is already the platform-blessed way on Raspberry Pi and ships with Raspberry Pi OS. ```python from gpiozero import MCP3008 import math # Defaults for a common 10k NTC thermistor + 10k divider, Vref = 3.3V. # Adjust BETA, T0, R0, R_FIXED to match your actual part's datasheet. def read_celsius(channel=0, beta=3950, t0=25.0, r0=10000.0, r_fixed=10000.0): adc = MCP3008(channel=channel) ratio = adc.value # 0.0..1.0 fraction of Vref if ratio <= 0.0 or ratio >= 1.0: raise ValueError(f"ADC reading {ratio} at rail; check wiring") # Thermistor on top of divider, fixed resistor to ground: # ratio = R_fixed / (R_therm + R_fixed) r_therm = r_fixed * (1.0 / ratio - 1.0) t0_k = t0 + 273.15 # Beta equation: 1/T = 1/T0 + (1/beta) * ln(R/R0) inv_t = 1.0 / t0_k + math.log(r_therm / r0) / beta return 1.0 / inv_t - 273.15 ``` Self-check (runs without hardware): ```python if __name__ == "__main__": # At ratio 0.5, R_therm == R0 == 10k, so T should read ~T0 (25 C). import unittest.mock as m with m.patch(__name__ + ".MCP3008") as fake: fake.return_value.value = 0.5 t = read_celsius() assert abs(t - 25.0) < 0.1, t print("ok") ``` What I skipped / risks: - The thermistor constants (BETA, R0, R_fixed, divider topology, Vref) are **guesses for a generic 10k NTC** — wrong values give wrong temperatures. Plug in your actual part's datasheet and confirm whether the thermistor sits on the Vref side or the ground side of the divider; the formula assumes thermistor-to-Vref. - No averaging/smoothing — a single raw sample is noisy; add a few-sample median if you need stable readings. - Didn't verify `gpiozero` is installed on your image (it is by default on Raspberry Pi OS); if not, `pip install gpiozero`.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main