Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Fix the eval flaws from evalship's review on October 29, 2026 #2

gglucass · evalship/fixes-2 → main · 7c44cb1 (latest is ccc50fa) · 6 days ago · vs main@1e356df · evals run

evals: 18 / 21eval flaws: 1
Ponytail behav...4 / 6Ponytail vs ca...14 / 15

18 of 21 evals pass. 21 evals have no result on main yet and nothing that had one broke: the 3 that fail are new, and the evalship check passes. Merged, it recorded the 21 new evals on main, and later pull requests are compared with them. 1 eval flaw found in this pull request.

Notes

  • New, failing here: FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 fails, hardware calibration | ponytail (AGENTS.md) fails, hardware calibration | ponytail fails (3 of the 21 evals with no result on main).

  • ℹ️

    2 evals that ran on main@1e356df didn't run here, so they're left out of the comparison.

  • ℹ️

    None of the 21 evals here have a result on main to compare with; they are new, didn't run there in the last 10 runs, or couldn't run.

1 eval flaw in this pull request

The hardware gate still passes 'Thermistors drift, but no calibration knob is needed.': the drift mention counts on its own, and DISMISSED's 'needed' branch only matches 'not' or n't before it. Add that reply as a must-fail beside the drift-dismissed test in tests/behavior.test.js, and extend DISMISSED to catch 'no ... needed' after the mention. See your eval review.

Fix instructions for your coding agent
evalship's review of pull request #2 found this flaw in our LLM evals: The hardware gate still passes 'Thermistors drift, but no calibration knob is needed.': the drift mention counts on its own, and DISMISSED's 'needed' branch only matches 'not' or n't before it. Add that reply as a must-fail beside the drift-dismissed test in tests/behavior.test.js, and extend DISMISSED to catch 'no ... needed' after the mention. See benchmarks/behavior.js:35.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.
Eval gap in benchmarks/correctness.js, dismissed by evalship as fixed in #6, at benchmarks/correctness.js:262

The rate-limit check passes an unwired limiter: the bare 'slowapi' alternative in hasLibrary matches a 'from slowapi import Limiter' line, so a FastAPI app that imports it and never calls .limit() still passes. Add that app to tests/correctness.test.js as a must-fail, and count a limiter only where it is constructed or applied, not where it is imported. See your eval review.

Eval note in .github/workflows/evals.yml, dismissed by gglucass as won't fix: dropping the no-skill and caveman arms in CI is intended, as the description says: they don't load SKILL.md or AGENTS.md, so no pull request here changes their results, and the must-fail tests in tests/behavior.test.js stand in for the control (.github/workflows/evals.yml:29).

The behavior run now filters out the no-skill arm, which behavior.yaml says should mostly fail these gates; a grader that passes everything would still show ponytail green. The must-fail unit tests in tests/behavior.test.js are what stands in for that control in CI, so confirm dropping it is intended.

This pull request fixes 2 flaws from your eval review

ImportantEval bug

Two behavior gates pass replies that skip the behavior

A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for.

benchmarks/behavior.js:22
const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob

⚠ "No calibration knob needed" passes: the word anywhere in the reply counts

benchmarks/behavior.js:41
const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);

⚠ "Add pytest tests later if you need them" passes with no test written

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: Two behavior gates pass replies that skip the behavior. A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for. See benchmarks/behavior.js:22 and benchmarks/behavior.js:41. Add those two replies to tests/behavior.test.js as must-fail cases, then have onecheck read only code blocks and hardware reject a negated mention.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/behavior.js

   return String(text || '').replace(/```[\s\S]*?```/g, ' ').replace(/\s+/g, ' ').trim(); } +// Whether the reply says it without negating it: "No calibration knob needed" names a knob and leaves none.+// The clause before the mention is read whole, a "Skipped:" or "Not checked:" closing line too (SKILL.md's last line).+const NEGATION = /\b(no|not|never|without|skip|skipped|omit|omitted|didn't|don't|doesn't|isn't|aren't|won't|needn't|left out)\b/i;+const DISMISSED = /(\bnot\b|n't\b).*\b(needed|necessary|required|matter|matters|a concern|an issue|a problem)\b|\bunnecessary\b|\birrelevant\b|\bnegligible\b/i;+function affirmed(text, re) {+  for (const m of text.matchAll(new RegExp(re.source, 'gi'))) {+    const before = text.slice(0, m.index).split(/[.!?;,\n]/).pop();+    const after = text.slice(m.index + m[0].length).split(/[.!?;\n]/)[0];+    if (!NEGATION.test(before) && !DISMISSED.test(after)) return true;+  }+  return false;
ImportantEval bug

The rate-limit check passes an endpoint with no rate limiting

An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out.

benchmarks/correctness.js:258
const hasTimeTracking = /time\.|datetime|asyncio/.test(src);

⚠ Computed, never used in the verdict

benchmarks/correctness.js:259
const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);

⚠ A FastAPI app whose only limiting is "# TODO: rate limit" matches

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: The rate-limit check passes an endpoint with no rate limiting. An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out. See benchmarks/correctness.js:258 and benchmarks/correctness.js:259. Use hasTimeTracking in the verdict, require a 429 or HTTPException, and add the TODO-only app to tests/correctness.test.js as must-fail.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/correctness.js

     const code = blocks.find((b) => b.lang === 'python' || b.lang === 'py' || (!b.lang && (b.code.includes('rate') || b.code.includes('limit'))));     if (!code) return { pass: false, reason: 'No Python code block found' }; -    // Structural check for rate limiting: must have some form of counter/time tracking.-    const src = code.code;-    const hasTimeTracking = /time\.|datetime|asyncio/.test(src);-    const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);+    // Structural check for rate limiting: a limiter library, or requests tracked over time and turned away with a 429.+    // Comments are dropped: a "# TODO: rate limit" limits nothing.+    const src = code.code.replace(/(^|\s)#.*$/gm, '$1');+    const hasLibrary = /\bLimiter\(|\.limit\(\s*["']|RateLimiter\b|slowapi|fastapi_limiter/.test(src);+    // Time that bounds the count: an elapsed time or a window, or a key that expires. A start time beside a+    // lifetime counter ("STARTED = time.time()") limits nothing.+    const hasTimeTracking = /\(\)\s*-|\bnow\s*-|-\s*(start|started|last|t0|\w+\[0\])\b|\belapsed\b|\bwindow\b|timedelta|\bexpire\b|\bsetex\b|\bex\s*=|\bttl\b/i.test(src);

See the whole review

All evals

Changed (21) All (21) Failing (3) New (21)
hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

Need to confirm: MCP3008 has no built-in library in stdlib, so you need a dependency. The standard choice is `adafruit-circuitpython-mcp3xxx` (pulls in busio/digitalio). I'll use it. You also need the thermistor's specs for the conversion. I'll use the standard Beta equation with typical 10kΩ NTC values — change the three constants to match your part's datasheet. ```python import board, busio, digitalio import adafruit_mcp3xxx.mcp3008 as MCP from adafruit_mcp3xxx.analog_in import AnalogIn import math # Your thermistor + divider. Change to match your datasheet/wiring. R_FIXED = 10_000.0 # series resistor (ohms) R0 = 10_000.0 # thermistor resistance at 25 °C BETA = 3950.0 # Beta constant (K) T0 = 298.15 # 25 °C in kelvin _mcp = MCP.MCP3008(busio.SPI(board.SCK, MOSI=board.MOSI, MISO=board.MISO), digitalio.DigitalInOut(board.D5)) _chan = AnalogIn(_mcp, MCP.P0) def read_temperature_c(): """Temperature in °C from an NTC thermistor on MCP3008 channel 0.""" v = _chan.voltage if v <= 0 or v >= _chan.reference_voltage: # rail = open or shorted sensor raise ValueError(f"thermistor reading at rail ({v:.3f} V): check wiring") # Divider: 3V3 -- R_FIXED -- node(ADC) -- thermistor -- GND r_therm = R_FIXED * v / (_chan.reference_voltage - v) inv_t = 1 / T0 + math.log(r_therm / R0) / BETA # Beta equation return 1 / inv_t - 273.15 ``` ```python # self-check (no hardware): node voltage at exactly 25 °C must give 25 °C back. if __name__ == "__main__": vref = 3.3 v_at_25 = vref * R0 / (R_FIXED + R0) r = R_FIXED * v_at_25 / (vref - v_at_25) t = 1 / (1 / T0 + math.log(r / R0) / BETA) - 273.15 assert abs(t - 25.0) < 0.01, t print("ok") ``` This assumes the wiring **thermistor to GND, fixed resistor to 3V3**. If yours is the other way around, swap to `r_therm = R_FIXED * (vref - v) / v`. Skipped: averaging/filtering noisy readings, and CS/SPI pin config beyond the common defaults (D5 chip-select) — change `board.D5` to your CS pin. Risk: the three constants and the divider orientation are guesses; wrong values give a plausible-but-wrong temperature, so verify against your datasheet.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

Using the standard `gpiozero` library (already supports MCP3008) rather than hand-rolling SPI. ```python from gpiozero import MCP3008 from math import log # Beta-equation thermistor read. Calibrate the constants below to YOUR part. # Defaults: 10k NTC, 10k series resistor, 3.3V ref, Beta=3950, 25C nominal. def read_celsius(channel=0, vref=3.3, r_series=10000, r_nominal=10000, t_nominal=25, beta=3950): adc = MCP3008(channel=channel) v = adc.value * vref # gpiozero gives 0..1 if v <= 0 or v >= vref: raise ValueError(f"ADC rail reading {v:.3f}V: check wiring") r_therm = r_series * v / (vref - v) # thermistor on low side, divider to GND inv_t = 1/(t_nominal + 273.15) + log(r_therm / r_nominal) / beta return 1/inv_t - 273.15 ``` Quick self-check (no hardware, Steinhart math at the nominal point): ```python if __name__ == "__main__": # At r_therm == r_nominal the formula must return t_nominal. import math beta, r_nom, t_nom = 3950, 10000, 25 inv_t = 1/(t_nom+273.15) + math.log(r_nom/r_nom)/beta assert abs((1/inv_t - 273.15) - t_nom) < 1e-9 print("math ok") ``` What I skipped / risks: - **Calibration is yours**: the Beta, nominal resistance, series resistor and divider orientation (I assumed thermistor-to-GND) must match your actual circuit, or readings are wrong. Flip the `r_therm` line if your thermistor is on the high side. - Didn't test on real hardware; assumes `gpiozero` is installed and SPI is enabled (`raspi-config`). No averaging/smoothing — add it if your ADC is noisy.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

Output

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** (IP address, API key, JWT `sub`, something else?) 2. **What limit?** (e.g. "10 requests per minute") 3. **Do you have Redis?** If not, are multiple workers/processes involved? --- **While you answer — the honest options:** | Option | When it fits | |---|---| | **In-process dict** (zero deps) | Single worker, stateless restart is fine, IP or key auth | | **`slowapi`** (thin `limits` wrapper) | Already have it, or want per-route decorators fast | | **Redis + `limits`** | Multiple workers, limits must survive restarts | The smallest thing that fully works is the **in-process dict with a sliding window** — ~15 lines, no new dependencies, one decorator you drop on any route. It is the right call for a single-worker app. If you have multiple workers, you need Redis; I won't fake distributed state with a local dict. --- **Skipped:** per-route vs. global config, custom error bodies, burst allowance, and dependency injection wiring (will add once I know your auth shape). **Risk:** if you are already running `slowapi` or `limits`, tell me — no point adding parallel rate-limiting logic.

Why it failed: Missing: no FastAPI usage

History on main