Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Fix the eval flaws from evalship's review on October 29, 2026 #2

gglucass · evalship/fixes-2 → main · 60f53b7 (latest is ccc50fa) · 6 days ago · vs main@1e356df · evals run

evals: 20 / 21eval flaws: 2
Ponytail behav...5 / 6Ponytail vs ca...15 / 15

20 of 21 evals pass. 21 evals have no result on main yet and nothing that had one broke: the 1 that fails is new, and the evalship check passes. Merged, it recorded the 21 new evals on main, and later pull requests are compared with them. 2 eval flaws found in this pull request.

Notes

  • New, failing here: hardware calibration | ponytail (AGENTS.md) fails (1 of the 21 evals with no result on main).

  • ℹ️

    2 evals that ran on main@1e356df didn't run here, so they're left out of the comparison.

  • ℹ️

    None of the 21 evals here have a result on main to compare with; they are new, didn't run there in the last 10 runs, or couldn't run.

2 eval flaws in this pull request

A bare unfenced answer with an assert now fails the one-check gate, since only fenced blocks are read; correctness.js falls back to the whole reply because terse models often skip fences. When no fence is present, score the whole reply with comments removed and add a must-pass unfenced case plus a must-fail prose-only case to tests/behavior.test.js.

Fix instructions for your coding agent
evalship's review of pull request #2 found this flaw in our LLM evals: A bare unfenced answer with an assert now fails the one-check gate, since only fenced blocks are read; correctness.js falls back to the whole reply because terse models often skip fences. When no fence is present, score the whole reply with comments removed and add a must-pass unfenced case plus a must-fail prose-only case to tests/behavior.test.js. See benchmarks/behavior.js:54.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.

Replies like 'Per-unit drift doesn't matter here' still pass the hardware gate: DISMISSED's after-clause lacks matter, problem and doesn't, which its n't alternative can't match. Extend DISMISSED with those words and add that reply as a must-fail case in tests/behavior.test.js. See your eval review.

Fix instructions for your coding agent
evalship's review of pull request #2 found this flaw in our LLM evals: Replies like 'Per-unit drift doesn't matter here' still pass the hardware gate: DISMISSED's after-clause lacks matter, problem and doesn't, which its n't alternative can't match. Extend DISMISSED with those words and add that reply as a must-fail case in tests/behavior.test.js. See benchmarks/behavior.js:20.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.
Eval gap in benchmarks/correctness.js, dismissed by evalship as fixed in #6, at benchmarks/correctness.js:262

A lifetime counter passes whenever a time module call appears, e.g. 'STARTED = time.time()' beside a global count that raises HTTPException(429), because the time and 429 checks are independent matches. Require the time to bound the count (a window or Redis expiry) and add this case to tests/correctness.test.js. See your eval review.

This pull request fixes 2 flaws from your eval review

ImportantEval bug

Two behavior gates pass replies that skip the behavior

A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for.

benchmarks/behavior.js:22
const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob

⚠ "No calibration knob needed" passes: the word anywhere in the reply counts

benchmarks/behavior.js:41
const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);

⚠ "Add pytest tests later if you need them" passes with no test written

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: Two behavior gates pass replies that skip the behavior. A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for. See benchmarks/behavior.js:22 and benchmarks/behavior.js:41. Add those two replies to tests/behavior.test.js as must-fail cases, then have onecheck read only code blocks and hardware reject a negated mention.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/behavior.js

   return String(text || '').replace(/```[\s\S]*?```/g, ' ').replace(/\s+/g, ' ').trim(); } +// Whether the reply says it without negating it: "No calibration knob needed" names a knob and leaves none.+// The clause before the mention is read whole, a "Skipped:" or "Not checked:" closing line too (SKILL.md's last line).+const NEGATION = /\b(no|not|never|without|skip|skipped|omit|omitted|didn't|don't|doesn't|isn't|aren't|won't|needn't|left out)\b/i;+const DISMISSED = /\b(not|isn't|aren't|n't)\b.*\b(needed|necessary|required|a concern|an issue)\b|\bunnecessary\b/i;+function affirmed(text, re) {+  for (const m of text.matchAll(new RegExp(re.source, 'gi'))) {+    const before = text.slice(0, m.index).split(/[.!?;,\n]/).pop();+    const after = text.slice(m.index + m[0].length).split(/[.!?;\n]/)[0];+    if (!NEGATION.test(before) && !DISMISSED.test(after)) return true;+  }+  return false;
ImportantEval bug

The rate-limit check passes an endpoint with no rate limiting

An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out.

benchmarks/correctness.js:258
const hasTimeTracking = /time\.|datetime|asyncio/.test(src);

⚠ Computed, never used in the verdict

benchmarks/correctness.js:259
const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);

⚠ A FastAPI app whose only limiting is "# TODO: rate limit" matches

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: The rate-limit check passes an endpoint with no rate limiting. An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out. See benchmarks/correctness.js:258 and benchmarks/correctness.js:259. Use hasTimeTracking in the verdict, require a 429 or HTTPException, and add the TODO-only app to tests/correctness.test.js as must-fail.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/correctness.js

     const code = blocks.find((b) => b.lang === 'python' || b.lang === 'py' || (!b.lang && (b.code.includes('rate') || b.code.includes('limit'))));     if (!code) return { pass: false, reason: 'No Python code block found' }; -    // Structural check for rate limiting: must have some form of counter/time tracking.-    const src = code.code;-    const hasTimeTracking = /time\.|datetime|asyncio/.test(src);-    const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);+    // Structural check for rate limiting: a limiter library, or requests tracked over time and turned away with a 429.+    // Comments are dropped: a "# TODO: rate limit" limits nothing.+    const src = code.code.replace(/(^|\s)#.*$/gm, '$1');+    const hasLibrary = /\bLimiter\(|\.limit\(\s*["']|RateLimiter\b|slowapi|fastapi_limiter/.test(src);+    const hasTimeTracking = /time\.|\btime\(\)|datetime|asyncio|monotonic|perf_counter|\bexpire\b|\bsetex\b|\bex\s*=|\bttl\b/i.test(src);+    const rejects = /429|Too Many|TOO_MANY_REQUESTS|HTTPException/.test(src);     const hasFastAPI = /fastapi|FastAPI|app\s*=|@app\./.test(src);

See the whole review

All evals

Changed (21) All (21) Failing (1) New (21)
hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

Reasonable laziness: use the standard library for MCP3008 reading via `gpiozero`, which is already the platform-blessed way on Raspberry Pi and ships with Raspberry Pi OS. ```python from gpiozero import MCP3008 import math # Defaults for a common 10k NTC thermistor + 10k divider, Vref = 3.3V. # Adjust BETA, T0, R0, R_FIXED to match your actual part's datasheet. def read_celsius(channel=0, beta=3950, t0=25.0, r0=10000.0, r_fixed=10000.0): adc = MCP3008(channel=channel) ratio = adc.value # 0.0..1.0 fraction of Vref if ratio <= 0.0 or ratio >= 1.0: raise ValueError(f"ADC reading {ratio} at rail; check wiring") # Thermistor on top of divider, fixed resistor to ground: # ratio = R_fixed / (R_therm + R_fixed) r_therm = r_fixed * (1.0 / ratio - 1.0) t0_k = t0 + 273.15 # Beta equation: 1/T = 1/T0 + (1/beta) * ln(R/R0) inv_t = 1.0 / t0_k + math.log(r_therm / r0) / beta return 1.0 / inv_t - 273.15 ``` Self-check (runs without hardware): ```python if __name__ == "__main__": # At ratio 0.5, R_therm == R0 == 10k, so T should read ~T0 (25 C). import unittest.mock as m with m.patch(__name__ + ".MCP3008") as fake: fake.return_value.value = 0.5 t = read_celsius() assert abs(t - 25.0) < 0.1, t print("ok") ``` What I skipped / risks: - The thermistor constants (BETA, R0, R_fixed, divider topology, Vref) are **guesses for a generic 10k NTC** — wrong values give wrong temperatures. Plug in your actual part's datasheet and confirm whether the thermistor sits on the Vref side or the ground side of the divider; the formula assumes thermistor-to-Vref. - No averaging/smoothing — a single raw sample is noisy; add a few-sample median if you need stable readings. - Didn't verify `gpiozero` is installed on your image (it is by default on Raspberry Pi OS); if not, `pip install gpiozero`.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

```python import spidev # Beta-model constants for a 10k NTC thermistor + 10k series resistor @ 3.3V. # Adjust BETA, T0, R0, R_SERIES, V_REF to match your part/wiring. import math BETA, T0, R0, R_SERIES, V_REF = 3950.0, 298.15, 10000.0, 10000.0, 3.3 def read_celsius(spi, channel=0): r = spi.xfer2([1, (8 + channel) << 4, 0]) adc = ((r[1] & 3) << 8) | r[2] # 10-bit result, 0..1023 if adc == 0 or adc >= 1023: raise ValueError(f"ADC out of range ({adc}): check wiring") v = adc / 1023.0 * V_REF r_therm = R_SERIES * v / (V_REF - v) # thermistor on top, series to GND inv_t = 1/T0 + math.log(r_therm / R0) / BETA return 1/inv_t - 273.15 if __name__ == "__main__": spi = spidev.SpiDev() spi.open(0, 0) spi.max_speed_hz = 1_000_000 print(f"{read_celsius(spi):.1f} °C") spi.close() ``` Self-check (no hardware needed): ```python if __name__ != "__main__": pass # quick sanity: at R_therm == R0 the temp must equal T0 (25 °C) class _Fake: def xfer2(self, _): return [0, 1, 0xFF] # adc = 511 ≈ midpoint → ~25 °C assert abs(read_celsius(_Fake()) - 25.0) < 1.0 ``` I used the Beta equation (not Steinhart-Hart) and took `spi` as a parameter instead of hiding global setup — one object, reusable, testable. Skipped: averaging/smoothing, and I assumed thermistor-to-3.3V / resistor-to-GND wiring. If yours is flipped, swap the `r_therm` formula. You **must** set `BETA`/`R0`/`R_SERIES`/`V_REF` to your actual parts or the reading is meaningless.

History on main

one runnable check | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / one runnable check | ponytail New

Input

{"probe":"onecheck","task":"Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."}

Output

```python import re def parse_duration(s): units = {'h': 3600, 'm': 60, 's': 1} if not re.fullmatch(r'(\d+[hms])+', s): raise ValueError(f'invalid duration: {s!r}') return sum(int(n) * units[u] for n, u in re.findall(r'(\d+)([hms])', s)) assert parse_duration('1h30m45s') == 5445 assert parse_duration('45s') == 45 assert parse_duration('2h') == 7200 ``` Skipped: days/ms, fractional values ('1.5h'), and unit-order enforcement ('30m1h' is accepted) — none were asked for. Risk: empty string raises ValueError, which is likely what you want.

History on main

one runnable check | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / one runnable check | ponytail (AGENTS.md) New

Input

{"probe":"onecheck","task":"Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."}

Output

```python import re def parse_duration(s): units = {'h': 3600, 'm': 60, 's': 1} matches = re.findall(r'(\d+)([hms])', s) if not matches or ''.join(n + u for n, u in matches) != s: raise ValueError(f"invalid duration: {s!r}") return sum(int(n) * units[u] for n, u in matches) assert parse_duration('1h30m45s') == 5445 assert parse_duration('45s') == 45 assert parse_duration('2h') == 7200 try: parse_duration('1x'); assert False except ValueError: pass ``` The `!= s` check rejects garbage like `'1x'` or `'1h foo'` instead of silently parsing part of it. Skipped: fractional/negative values, days/weeks, and whitespace tolerance — none were asked for. Risk: unit order isn't enforced, so `'45s1h'` parses to 3645; add order checking if your inputs need strict `h→m→s` ordering.

History on main

requested explanation | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / requested explanation | ponytail New

Input

{"probe":"explanation","task":"Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] \u003e 0:\n r.append(x['a'] * 2)\n return r"}

Output

Renamed for readability, kept the logic identical: ```python def doubled_positives(items): return [item['a'] * 2 for item in items if item.get('a', 0) > 0] ``` Changes: - `p` → `doubled_positives`, `d` → `items`, `x` → `item`: names say what they hold. - Collapsed the loop into a list comprehension — same output, less noise. - `x.get('a') and x['a'] > 0` → `item.get('a', 0) > 0`: one lookup instead of two; the `> 0` already rejects `None`/`0`/missing. You asked for a *detailed step-by-step write-up*, but there's nothing more to say than the three lines above — the function is four lines. A long essay would be padding. One behavior note: the original `x.get('a') and ...` also skipped non-numeric truthy values differently; my version assumes `'a'` is always a number or absent. If `'a'` can be a non-number, say so and I'll guard it. Didn't add a test (trivial transform) or handle non-numeric `'a'` — flag if either matters.

History on main

requested explanation | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / requested explanation | ponytail (AGENTS.md) New

Input

{"probe":"explanation","task":"Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] \u003e 0:\n r.append(x['a'] * 2)\n return r"}

Output

Here's the readable version: ```python def doubled_positive_amounts(items): return [item['a'] * 2 for item in items if item.get('a', 0) > 0] ``` Changes: 1. **`p` → `doubled_positive_amounts`** — the name now says what it returns. 2. **`d` → `items`, `x` → `item`** — meaningful names for the collection and its elements. 3. **Loop + `append` → list comprehension** — same result, one line, and the standard Python idiom for "build a list by filtering and mapping." 4. **`x.get('a') and x['a'] > 0` → `item.get('a', 0) > 0`** — the original used `.get('a')` to guard against a missing/falsy key, then re-indexed `x['a']`. `get('a', 0)` does both in one read: missing key defaults to `0`, which fails `> 0`. Behavior note below. One behavior difference worth flagging: - Original: `x.get('a')` is falsy for `None`, `0`, `''`, `[]`, etc., short-circuiting before the comparison. - New: `get('a', 0) > 0` only substitutes `0` for a *missing* key. If a value is `None`, the original silently skipped it (`None` is falsy); the new version raises `TypeError` on `None > 0`. If `'a'` can be `None`, keep the explicit guard instead: ```python def doubled_positive_amounts(items): return [item['a'] * 2 for item in items if (item.get('a') or 0) > 0] ``` `(item.get('a') or 0)` maps missing *and* `None`/falsy to `0`, matching the original's skip behavior exactly. --- I didn't check what values `'a'` actually holds in your data, so I gave you both versions. Pick the `or 0` one if `None`/non-numeric values can appear; otherwise the first is cleaner. Risk: the first version changes behavior (raises) on `None` values where the original silently skipped them.

History on main

CSV sum | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

Output

```python import csv total = sum(float(row['amount']) for row in csv.DictReader(open('sales.csv'))) print(total) ``` **Skipped:** error handling (file missing, bad data), context manager (fine for a script). **Risk:** if 'amount' has non-numeric values, it crashes—add `.strip()` and a try/except if the data is messy.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

Output

```python import csv with open("sales.csv", newline="") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(total) ``` Skipped: error handling for missing file/column/bad numbers, and didn't check your CSV's actual column names or delimiter. Risk: crashes if `amount` has blanks or non-numeric values.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

Output

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` --- Skipped: error handling for missing file or bad values. Risk: crashes on empty `amount` cells or non-numeric data.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

Output

# Smallest complete solution Use `slowapi`, it's the FastAPI standard: ```bash pip install slowapi ``` ```python from slowapi import Limiter from slowapi.util import get_remote_address from fastapi import FastAPI limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exc_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") # or "100/hour", etc. async def your_endpoint(request: Request): return {"status": "ok"} ``` If you're behind a proxy (Heroku, load balancer), add to limiter init: ```python limiter = Limiter(key_func=get_remote_address, headers_use_forwarded_for=True) ``` **What I skipped:** per-user limits (needs auth context), distributed rate limiting across servers (use Redis), per-endpoint config files (hardcode limits, change via env vars if needed), fancy UI responses. **Risk:** If you're on a shared IP (corporate network, some clouds), all users hit the same limit. Pass `key_func=lambda r: r.headers.get("user-id")` if you have user IDs in headers, or use auth scopes if you have a login system.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

Output

Use `slowapi` (built on the standard `Limiter` pattern for FastAPI/Starlette). A few lines, no custom bucket logic: ```python # pip install slowapi from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded from fastapi import FastAPI, Request limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") # tune the number async def your_endpoint(request: Request): # `request` param is required ... ``` Two things that bite people: - The endpoint **must** take `request: Request` or slowapi can't read the client IP. - Default storage is in-memory, so limits reset on restart and aren't shared across workers/replicas. If you run more than one worker, pass `storage_uri="redis://..."` to `Limiter`. Skipped: didn't add Redis (your in-memory default is fine for a single process). Didn't see your actual endpoint code, so adjust the route and rate to match — and note IP-based limiting is per-IP, not per-user, so anyone behind a shared NAT shares the budget.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

Output

**What are you protecting, and what do you already have?** Before writing anything, I need two answers: 1. **Show me the endpoint** (the route function and any existing middleware/dependencies). 2. **What's your deployment?** Single process, multiple workers, multiple servers? The answer changes the solution completely: | Setup | Right tool | |---|---| | Single process | In-memory dict, zero deps | | Multiple workers (same machine) | Still in-memory *if* you can pin users to a worker, otherwise Redis | | Multiple servers | Redis (or a proxy like nginx/Caddy that rate-limits before FastAPI) | --- **If you're on a single process right now**, here's the smallest thing that works — a dependency you drop onto any route, no new packages: ```python from collections import defaultdict from time import time from fastapi import HTTPException, Request # shortcut: in-memory, resets on restart, not shared across workers _hits: dict[str, list[float]] = defaultdict(list) def rate_limit(max_calls: int = 10, window: int = 60): def dependency(request: Request): key = request.client.host now = time() hits = [t for t in _hits[key] if now - t < window] if len(hits) >= max_calls: raise HTTPException(status_code=429, detail="Too many requests") hits.append(now) _hits[key] = hits return dependency ``` ```python # on your route: @app.post("/your-endpoint", dependencies=[Depends(rate_limit(max_calls=10, window=60))]) async def your_endpoint(): ... ``` **If you're on multiple workers or servers**, say so and I'll show the Redis version using `slowapi` (already the FastAPI community's standard, one decorator, no boilerplate). --- **Skipped:** auth-based limiting (per user token vs per IP), custom retry-after headers, sliding vs fixed window distinction. **Risk:** IP-based limiting breaks if users share a NAT or proxy — all get blocked together.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

Output

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // only this fires after 300ms ``` **What I skipped:** cancel/flush methods, leading/trailing edge options, max-wait timer. Add them only if callers need them.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

Output

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js let calls = 0; const d = debounce(() => calls++, 50); d(); d(); d(); setTimeout(() => console.assert(calls === 1, `expected 1, got ${calls}`), 100); ``` Skipped: leading-edge, cancel/flush methods — not asked for. `this` and args are preserved so it works on methods and event handlers.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

Output

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js // rapid calls → only one fires let count = 0; const inc = debounce(() => count++, 50); inc(); inc(); inc(); setTimeout(() => console.assert(count === 1, "expected 1 call, got " + count), 100); ``` --- **Skipped:** leading-edge option, cancel/flush methods, max-wait — none were asked for. **Risk:** `this` binding relies on a normal `function`; arrow-function callers lose context (usually fine in modern code).

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

Output

# CountdownTimer ```jsx import { useState, useEffect } from 'react'; export default function CountdownTimer({ seconds = 60 }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { if (remaining <= 0) return; const id = setTimeout(() => setRemaining(remaining - 1), 1000); return () => clearTimeout(id); }, [remaining]); return <div>{remaining}s</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={30} /> ``` **What I skipped:** pause/resume (add a paused state + useEffect guard if needed), callbacks on completion (add `onComplete` prop), styling, or resetting the timer. Start with this; tell me what the timer needs to *do* when it hits zero or if the user should control it. **Risk:** If `seconds` prop changes mid-countdown, the timer restarts. Pass a new key to reset it intentionally, or add a useEffect to set initial state only on mount if that's wrong for your use case.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

Output

```jsx import { useEffect, useState } from "react"; export default function Countdown({ seconds }) { const [left, setLeft] = useState(seconds); useEffect(() => { setLeft(seconds); const id = setInterval(() => setLeft((s) => Math.max(0, s - 1)), 1000); return () => clearInterval(id); }, [seconds]); return <span>{left}</span>; } ``` Skipped: pause/reset/onComplete, formatting (mm:ss), styling — none were asked for. `setInterval` drifts a little over long runs; fine for a display timer, not a stopwatch.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

Output

Here's the smallest version that does the job: ```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { if (remaining <= 0) return; const id = setInterval(() => setRemaining((r) => r - 1), 1000); return () => clearInterval(id); }, [remaining]); return <div>{remaining}s</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={60} /> ``` When `remaining` hits 0 the interval stops. The effect re-runs each tick, which keeps drift from accumulating across re-renders (each interval is one second, then cleared). --- **Skipped:** pause/resume, reset, onComplete callback, formatting as MM:SS, styling — none were asked for. Add `onComplete` as a prop if you need a side-effect at zero. **Risk:** `seconds` prop changes after mount are ignored; if you need that, reset state in a second `useEffect` on `seconds`.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Write me a Python function that validates email addresses."}

Output

```python import re def is_valid_email(email): return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (rarely needed), DNS checks, SMTP verification, internationalized domains, edge cases like quoted strings—all overkill unless you said "production-grade" or "spec-compliant." **Risk:** This accepts `a@b.c` (which is technically valid but may not be what users intend). If you need stricter rules or to actually send mail, tell me the real constraint—usually it's "I need to send them a confirmation email anyway."

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Write me a Python function that validates email addresses."}

Output

```python import re _EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.match(email)) ``` Quick self-check: ```python assert is_valid_email("a@b.co") assert not is_valid_email("a@b") assert not is_valid_email("a b@c.com") assert not is_valid_email("a@@b.com") ``` Skipped: full RFC 5322 compliance — no regex gets that right, and the only real validation is sending a confirmation email. shortcut: rejects quoted locals and IP-literal domains; upgrade to the `email-validator` package if you need those or deliverability checks.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Write me a Python function that validates email addresses."}

Output

```python import re def is_valid_email(email: str) -> bool: """Returns True if email looks valid enough for registration forms.""" pattern = r'^[^@\s]+@[^@\s]+\.[^@\s]+$' return bool(re.match(pattern, email)) ``` **Usage** ```python is_valid_email("user@example.com") # True is_valid_email("bad@") # False ``` --- **Skipped:** RFC 5322 edge cases (quoted locals, IP literals, subdomains-only). If you're validating for anything beyond "did the user type something email-shaped", send a confirmation email instead — that's the only real check.

Why it failed: All assertions passed

History on main