Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Fix the eval flaws from evalship's review on October 29, 2026 #2

gglucass · evalship/fixes-2 → main · ccc50fa · 6 days ago · vs main@1e356df · evals run

evals: 19 / 21eval flaws: 0
Ponytail behav...4 / 6Ponytail vs ca...15 / 15

19 of 21 evals pass. 21 evals have no result on main yet and nothing that had one broke: the 2 that fail are new, and the evalship check passes. Merged, it recorded the 21 new evals on main, and later pull requests are compared with them. No eval flaws found in this pull request.

Notes

  • New, failing here: hardware calibration | ponytail fails, hardware calibration | ponytail (AGENTS.md) fails (2 of the 21 evals with no result on main).

  • ℹ️

    2 evals that ran on main@1e356df didn't run here, so they're left out of the comparison.

  • ℹ️

    None of the 21 evals here have a result on main to compare with; they are new, didn't run there in the last 10 runs, or couldn't run.

0 eval flaws in this pull request

Eval gap in benchmarks/correctness.js, dismissed by evalship as fixed in #6, at benchmarks/correctness.js:262

The rate-limit time check matches bare words (window, ttl, elapsed) inside strings, so a counter with no time that returns 429 with 'this window' in its message passes. Add that reply to tests/correctness.test.js as a must-fail case, and strip string literals before matching time.

This pull request fixes 2 flaws from your eval review

ImportantEval bug

Two behavior gates pass replies that skip the behavior

A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for.

benchmarks/behavior.js:22
const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob

⚠ "No calibration knob needed" passes: the word anywhere in the reply counts

benchmarks/behavior.js:41
const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);

⚠ "Add pytest tests later if you need them" passes with no test written

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: Two behavior gates pass replies that skip the behavior. A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for. See benchmarks/behavior.js:22 and benchmarks/behavior.js:41. Add those two replies to tests/behavior.test.js as must-fail cases, then have onecheck read only code blocks and hardware reject a negated mention.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/behavior.js

   return String(text || '').replace(/```[\s\S]*?```/g, ' ').replace(/\s+/g, ' ').trim(); } +// Whether the reply says it without negating it: "No calibration knob needed" names a knob and leaves none.+// The clause before the mention is read whole, a "Skipped:" or "Not checked:" closing line too (SKILL.md's last line).+const NEGATION = /\b(no|not|never|without|skip|skipped|omit|omitted|didn't|don't|doesn't|isn't|aren't|won't|needn't|left out)\b/i;+// After the mention: dismissed ("doesn't matter"), skipped, or "no <calibration word> ... needed", not "no library needed".+const DISMISSED = /(\bnot\b|n't\b).*\b(needed|necessary|required|matter|matters|a concern|an issue|a problem)\b|\bno\b.*\b(calibration|knob|offset|trim|tuning)\b.*\b(needed|necessary|required)\b|\b(skipped|omitted|left out)\b|\bunnecessary\b|\birrelevant\b|\bnegligible\b/i;+function affirmed(text, re) {+  for (const m of text.matchAll(new RegExp(re.source, 'gi'))) {+    const before = text.slice(0, m.index).split(/[.!?;,\n]/).pop();+    const after = text.slice(m.index + m[0].length).split(/[.!?;\n]/)[0];+    if (!NEGATION.test(before) && !DISMISSED.test(after)) return true;+  }
ImportantEval bug

The rate-limit check passes an endpoint with no rate limiting

An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out.

benchmarks/correctness.js:258
const hasTimeTracking = /time\.|datetime|asyncio/.test(src);

⚠ Computed, never used in the verdict

benchmarks/correctness.js:259
const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);

⚠ A FastAPI app whose only limiting is "# TODO: rate limit" matches

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: The rate-limit check passes an endpoint with no rate limiting. An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out. See benchmarks/correctness.js:258 and benchmarks/correctness.js:259. Use hasTimeTracking in the verdict, require a 429 or HTTPException, and add the TODO-only app to tests/correctness.test.js as must-fail.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/correctness.js

     const code = blocks.find((b) => b.lang === 'python' || b.lang === 'py' || (!b.lang && (b.code.includes('rate') || b.code.includes('limit'))));     if (!code) return { pass: false, reason: 'No Python code block found' }; -    // Structural check for rate limiting: must have some form of counter/time tracking.-    const src = code.code;-    const hasTimeTracking = /time\.|datetime|asyncio/.test(src);-    const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);+    // Structural check for rate limiting: a limiter library, or requests tracked over time and turned away with a 429.+    // Comments are dropped: a "# TODO: rate limit" limits nothing.+    const src = code.code.replace(/(^|\s)#.*$/gm, '$1');+    // A limiter applied to a route or as the app's default, not one only imported or constructed.+    const hasLibrary = /@\s*\w*limiter\.limit\(|Depends\(\s*RateLimiter\(|default_limits\s*=/i.test(src);+    // Time that bounds the count: an elapsed time or a window, or a key that expires. A start time beside a+    // lifetime counter ("STARTED = time.time()") limits nothing.

See the whole review

All evals

Changed (21) All (21) Failing (2) New (21)
hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

```python from gpiozero import MCP3008 from math import log # Thermistor: 10k NTC, B=3950, in a divider with a 10k fixed resistor to 3.3V. # shortcut: constants hardcoded for a 10k/B3950 part; change if your thermistor differs. B = 3950 R_FIXED = 10_000 R0 = 10_000 T0 = 298.15 # 25°C in Kelvin def read_celsius(channel=0): adc = MCP3008(channel=channel) v = adc.value # 0.0–1.0 fraction of Vref if not 0.0 < v < 1.0: raise ValueError(f"ADC reading {v} out of usable range (open/short circuit?)") r_therm = R_FIXED * v / (1.0 - v) # thermistor on the ground side of the divider kelvin = 1.0 / (1.0 / T0 + log(r_therm / R0) / B) return kelvin - 273.15 ``` Uses `gpiozero`'s MCP3008 (already the standard on the Pi, handles SPI for you), so no new dependency and no raw bit-banging. Beta equation, not Steinhart-Hart — one coefficient, close enough for ambient sensing. Self-check (run off-Pi, no hardware): ```python def _check(): # at 25°C the divider sits at exactly half, v=0.5 -> 25.00°C assert abs((1.0 / (1.0/T0 + log(1.0)/B) - 273.15) - 25.0) < 1e-9 _check() ``` Skipped: no averaging/smoothing, no calibration offset, assumes 10k-NTC-to-ground wiring and 3.3V ref — flip the divider formula if your fixed resistor is on the ground side instead. Risk: wrong `B`/`R0` or divider orientation gives silently wrong temps, not errors — verify against a known reference once on real hardware.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

Using the standard `gpiozero` MCP3008 interface and the Steinhart-Hart equation (the calibration real thermistor hardware needs — a bare voltage-to-temp linear guess would be wrong). ```python from math import log from gpiozero import MCP3008 # Defaults for a common 10k NTC thermistor + 10k series resistor at 3.3V. # Beta model. Adjust R0/BETA/T0 to your part's datasheet. def read_celsius(channel=0, vref=3.3, r_series=10000.0, r0=10000.0, t0_c=25.0, beta=3950.0): adc = MCP3008(channel=channel) v = adc.value * vref # gpiozero gives 0..1 if v <= 0.0 or v >= vref: raise ValueError(f"thermistor voltage out of range: {v:.3f}V") # Divider: 3V3 -- r_series -- node(ADC) -- thermistor -- GND r_therm = r_series * v / (vref - v) # Beta equation (Steinhart-Hart simplified) t0 = t0_c + 273.15 inv_t = 1.0 / t0 + (1.0 / beta) * log(r_therm / r0) return 1.0 / inv_t - 273.15 ``` Self-check (no hardware needed — swap the ADC read for the math): ```python if __name__ == "__main__": # At r_therm == r0 the temp must equal t0_c. import math beta, r0, t0_c = 3950.0, 10000.0, 25.0 t0 = t0_c + 273.15 inv_t = 1.0 / t0 + (1.0 / beta) * math.log(10000.0 / r0) assert abs((1.0 / inv_t - 273.15) - 25.0) < 1e-6 print("ok") ``` Skipped: didn't add averaging/smoothing, async, or caching the MCP3008 object — add those if you poll in a tight loop. Risk: the divider wiring and the beta/R0 values are guesses from a common part; if your thermistor is on the **top** leg instead, swap `v` and `vref - v`, and set the three constants from your datasheet or the reading is meaningless.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main