Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Fix the eval flaws from evalship's review on October 29, 2026 #2

gglucass · evalship/fixes-2 → main · 8f5d960 (latest is ccc50fa) · 6 days ago · vs main@1e356df · evals run

evals: 19 / 21eval flaws: 1
Ponytail behav...5 / 6Ponytail vs ca...14 / 15

19 of 21 evals pass. 21 evals have no result on main yet and nothing that had one broke: the 2 that fail are new, and the evalship check passes. Merged, it recorded the 21 new evals on main, and later pull requests are compared with them. 1 eval flaw found in this pull request.

Notes

  • New, failing here: FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 fails, hardware calibration | ponytail fails (2 of the 21 evals with no result on main).

  • ℹ️

    2 evals that ran on main@1e356df didn't run here, so they're left out of the comparison.

  • ℹ️

    None of the 21 evals here have a result on main to compare with; they are new, didn't run there in the last 10 runs, or couldn't run.

1 eval flaw in this pull request

Eval note benchmarks/correctness.js:263

The rate-limit check now fails a fixed-window limiter keyed on int(time.time() // 60) with no expire, since time.time() isn't followed by a minus and no window term appears; the removed check passed that shape, though the description's rule ('tracked over time') accepts it. Add it as a must-pass test or confirm the stricter rule is intended.

Fix instructions for your coding agent 
evalship's review of pull request #2 found this flaw in our LLM evals: The rate-limit check now fails a fixed-window limiter keyed on `int(time.time() // 60)` with no `expire`, since `time.time()` isn't followed by a minus and no window term appears; the removed check passed that shape, though the description's rule ('tracked over time') accepts it. Add it as a must-pass test or confirm the stricter rule is intended. See benchmarks/correctness.js:263.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.
Eval gap in benchmarks/correctness.js, dismissed by evalship as fixed in #6, at benchmarks/correctness.js:262

The new rate-limit check accepts any RateLimiter( call, so one built but never applied to a route passes; the added unapplied-limiter test uses slowapi's Limiter, which that regex doesn't match. Require the limiter to be applied (Depends(RateLimiter( or a @...limit( decorator) and add the unapplied case as a must-fail test.

This pull request fixes 2 flaws from your eval review

ImportantEval bug

Two behavior gates pass replies that skip the behavior

A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for.

benchmarks/behavior.js:22
const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob

⚠ "No calibration knob needed" passes: the word anywhere in the reply counts

benchmarks/behavior.js:41
const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);

⚠ "Add pytest tests later if you need them" passes with no test written

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: Two behavior gates pass replies that skip the behavior. A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for. See benchmarks/behavior.js:22 and benchmarks/behavior.js:41. Add those two replies to tests/behavior.test.js as must-fail cases, then have onecheck read only code blocks and hardware reject a negated mention.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/behavior.js

   return String(text || '').replace(/```[\s\S]*?```/g, ' ').replace(/\s+/g, ' ').trim(); } +// Whether the reply says it without negating it: "No calibration knob needed" names a knob and leaves none.+// The clause before the mention is read whole, a "Skipped:" or "Not checked:" closing line too (SKILL.md's last line).+const NEGATION = /\b(no|not|never|without|skip|skipped|omit|omitted|didn't|don't|doesn't|isn't|aren't|won't|needn't|left out)\b/i;+// After the mention: dismissed ("doesn't matter"), skipped, or "no <calibration word> ... needed", not "no library needed".+const DISMISSED = /(\bnot\b|n't\b).*\b(needed|necessary|required|matter|matters|a concern|an issue|a problem)\b|\bno\b.*\b(calibration|knob|offset|trim|tuning)\b.*\b(needed|necessary|required)\b|\b(skipped|omitted|left out)\b|\bunnecessary\b|\birrelevant\b|\bnegligible\b/i;+function affirmed(text, re) {+  for (const m of text.matchAll(new RegExp(re.source, 'gi'))) {+    const before = text.slice(0, m.index).split(/[.!?;,\n]/).pop();+    const after = text.slice(m.index + m[0].length).split(/[.!?;\n]/)[0];+    if (!NEGATION.test(before) && !DISMISSED.test(after)) return true;+  }
ImportantEval bug

The rate-limit check passes an endpoint with no rate limiting

An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out.

benchmarks/correctness.js:258
const hasTimeTracking = /time\.|datetime|asyncio/.test(src);

⚠ Computed, never used in the verdict

benchmarks/correctness.js:259
const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);

⚠ A FastAPI app whose only limiting is "# TODO: rate limit" matches

Fix instructions for your coding agent 
Fix this flaw in our LLM evals, found by evalship's review: The rate-limit check passes an endpoint with no rate limiting. An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out. See benchmarks/correctness.js:258 and benchmarks/correctness.js:259. Use hasTimeTracking in the verdict, require a 429 or HTTPException, and add the TODO-only app to tests/correctness.test.js as must-fail.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/correctness.js

     const code = blocks.find((b) => b.lang === 'python' || b.lang === 'py' || (!b.lang && (b.code.includes('rate') || b.code.includes('limit'))));     if (!code) return { pass: false, reason: 'No Python code block found' }; -    // Structural check for rate limiting: must have some form of counter/time tracking.-    const src = code.code;-    const hasTimeTracking = /time\.|datetime|asyncio/.test(src);-    const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);+    // Structural check for rate limiting: a limiter library, or requests tracked over time and turned away with a 429.+    // Comments are dropped: a "# TODO: rate limit" limits nothing.+    const src = code.code.replace(/(^|\s)#.*$/gm, '$1');+    // A limiter applied to a route or as the app's default, not one only imported or constructed.+    const hasLibrary = /@\s*\w*limiter\.limit\(|RateLimiter\(|default_limits\s*=/i.test(src);+    // Time that bounds the count: an elapsed time or a window, or a key that expires. A start time beside a+    // lifetime counter ("STARTED = time.time()") limits nothing.

See the whole review

All evals

Changed (21) All (21) Failing (2) New (21)
hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

Need to know your thermistor's specs to convert resistance→temperature. Assuming a standard 10k NTC with a 10k series resistor and a Beta value — the common hobbyist setup. Using the `adafruit-circuitpython-mcp3008` lib (the standard way on Pi). ```python import math import board, busio, digitalio import adafruit_mcp3008 # Your thermistor's datasheet values — CHANGE THESE: BETA = 3950 # Beta coefficient (K) R0 = 10000 # thermistor resistance at 25°C (ohms) T0 = 298.15 # 25°C in kelvin R_SERIES = 10000 # fixed series resistor (ohms) def read_celsius(): spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) mcp = adafruit_mcp3008.MCP3008(spi, cs) raw = mcp.read(0) # 0..1023 on channel 0 if raw <= 0 or raw >= 1023: raise ValueError(f"ADC reading {raw} out of range — check wiring") # Thermistor on top of divider, series resistor to ground: # R_therm = R_SERIES * raw / (1023 - raw) r_therm = R_SERIES * raw / (1023 - raw) # Beta equation temp_k = 1 / (1 / T0 + math.log(r_therm / R0) / BETA) return temp_k - 273.15 ``` ```python # self-check (no hardware): raw=512 ~= R0 ~= 25°C class _Fake: read = lambda s, c: 512 import sys; sys.modules.setdefault("test", None) _t = 1/(1/298.15 + math.log(10000/R0)/BETA) - 273.15 assert abs(_t - 25) < 0.5, _t ``` **Skipped:** repeated-read averaging, keeping the SPI/MCP object alive across calls (reopening every call is wasteful — hoist it out if you poll often), and Steinhart-Hart (Beta is less accurate at temperature extremes). **You must know:** the four constants and the divider orientation (thermistor-to-3.3V vs thermistor-to-ground) are guesses — set them from your datasheet and wiring or the reading will be wrong, not just inaccurate.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

Output

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify "a user"?** IP address, API key, JWT `sub`, something else? 2. **What's the limit?** (e.g. 10 requests/minute) 3. **Do you have Redis?** Or does in-process memory work (single process, no load balancer)? 4. **Show me the endpoint** — I'll write the smallest change that fits what you already have. --- **Why I'm asking instead of guessing:** The answer changes the solution completely: | Situation | Smallest solution | |-----------|-------------------| | Single process, IP-based | `slowapi` (2-line decorator, wraps `limits`) | | Already have Redis | `fastapi-limiter` or `slowapi` + Redis backend | | Already use a reverse proxy (nginx, Caddy, Cloudflare) | **Rate limit there instead — zero app code** | | Auth middleware already present | Piggyback on it, not a new layer | The proxy option is worth a second look — if you have nginx/Cloudflare in front, `limit_req` or a Cloudflare rule solves this with no app code and no new dependency.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429)

History on main