Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Fix the eval flaws from evalship's review on October 29, 2026 #2

gglucass · evalship/fixes-2 → main · 8f5d960 (latest is ccc50fa) · 6 days ago · vs main@1e356df · evals run

evals: 19 / 21eval flaws: 1
Ponytail behav...5 / 6Ponytail vs ca...14 / 15

19 of 21 evals pass. 21 evals have no result on main yet and nothing that had one broke: the 2 that fail are new, and the evalship check passes. Merged, it recorded the 21 new evals on main, and later pull requests are compared with them. 1 eval flaw found in this pull request.

Notes

  • New, failing here: FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 fails, hardware calibration | ponytail fails (2 of the 21 evals with no result on main).

  • ℹ️

    2 evals that ran on main@1e356df didn't run here, so they're left out of the comparison.

  • ℹ️

    None of the 21 evals here have a result on main to compare with; they are new, didn't run there in the last 10 runs, or couldn't run.

1 eval flaw in this pull request

The rate-limit check now fails a fixed-window limiter keyed on int(time.time() // 60) with no expire, since time.time() isn't followed by a minus and no window term appears; the removed check passed that shape, though the description's rule ('tracked over time') accepts it. Add it as a must-pass test or confirm the stricter rule is intended.

Fix instructions for your coding agent
evalship's review of pull request #2 found this flaw in our LLM evals: The rate-limit check now fails a fixed-window limiter keyed on `int(time.time() // 60)` with no `expire`, since `time.time()` isn't followed by a minus and no window term appears; the removed check passed that shape, though the description's rule ('tracked over time') accepts it. Add it as a must-pass test or confirm the stricter rule is intended. See benchmarks/correctness.js:263.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.
Eval gap in benchmarks/correctness.js, dismissed by evalship as fixed in #6, at benchmarks/correctness.js:262

The new rate-limit check accepts any RateLimiter( call, so one built but never applied to a route passes; the added unapplied-limiter test uses slowapi's Limiter, which that regex doesn't match. Require the limiter to be applied (Depends(RateLimiter( or a @...limit( decorator) and add the unapplied case as a must-fail test.

This pull request fixes 2 flaws from your eval review

ImportantEval bug

Two behavior gates pass replies that skip the behavior

A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for.

benchmarks/behavior.js:22
const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob

⚠ "No calibration knob needed" passes: the word anywhere in the reply counts

benchmarks/behavior.js:41
const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);

⚠ "Add pytest tests later if you need them" passes with no test written

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: Two behavior gates pass replies that skip the behavior. A SKILL.md change that stops the model leaving a calibration knob or a check still passes whenever the reply mentions what it left out, so the gates miss the regressions they exist for. See benchmarks/behavior.js:22 and benchmarks/behavior.js:41. Add those two replies to tests/behavior.test.js as must-fail cases, then have onecheck read only code blocks and hardware reject a negated mention.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/behavior.js

   return String(text || '').replace(/```[\s\S]*?```/g, ' ').replace(/\s+/g, ' ').trim(); } +// Whether the reply says it without negating it: "No calibration knob needed" names a knob and leaves none.+// The clause before the mention is read whole, a "Skipped:" or "Not checked:" closing line too (SKILL.md's last line).+const NEGATION = /\b(no|not|never|without|skip|skipped|omit|omitted|didn't|don't|doesn't|isn't|aren't|won't|needn't|left out)\b/i;+// After the mention: dismissed ("doesn't matter"), skipped, or "no <calibration word> ... needed", not "no library needed".+const DISMISSED = /(\bnot\b|n't\b).*\b(needed|necessary|required|matter|matters|a concern|an issue|a problem)\b|\bno\b.*\b(calibration|knob|offset|trim|tuning)\b.*\b(needed|necessary|required)\b|\b(skipped|omitted|left out)\b|\bunnecessary\b|\birrelevant\b|\bnegligible\b/i;+function affirmed(text, re) {+  for (const m of text.matchAll(new RegExp(re.source, 'gi'))) {+    const before = text.slice(0, m.index).split(/[.!?;,\n]/).pop();+    const after = text.slice(m.index + m[0].length).split(/[.!?;\n]/)[0];+    if (!NEGATION.test(before) && !DISMISSED.test(after)) return true;+  }
ImportantEval bug

The rate-limit check passes an endpoint with no rate limiting

An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out.

benchmarks/correctness.js:258
const hasTimeTracking = /time\.|datetime|asyncio/.test(src);

⚠ Computed, never used in the verdict

benchmarks/correctness.js:259
const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);

⚠ A FastAPI app whose only limiting is "# TODO: rate limit" matches

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: The rate-limit check passes an endpoint with no rate limiting. An answer that skips the limiter scores as correct and as fewer lines, which is the "less code because it did less" result the correctness gate is meant to rule out. See benchmarks/correctness.js:258 and benchmarks/correctness.js:259. Use hasTimeTracking in the verdict, require a 429 or HTTPException, and add the TODO-only app to tests/correctness.test.js as must-fail.
Fix it in a pull request; evalship confirms the fix on it.

✓ The fix, in benchmarks/correctness.js

     const code = blocks.find((b) => b.lang === 'python' || b.lang === 'py' || (!b.lang && (b.code.includes('rate') || b.code.includes('limit'))));     if (!code) return { pass: false, reason: 'No Python code block found' }; -    // Structural check for rate limiting: must have some form of counter/time tracking.-    const src = code.code;-    const hasTimeTracking = /time\.|datetime|asyncio/.test(src);-    const hasLimitLogic = /limit|max_requests|rate|429|Too Many|HTTPException|RateLimiter/.test(src);+    // Structural check for rate limiting: a limiter library, or requests tracked over time and turned away with a 429.+    // Comments are dropped: a "# TODO: rate limit" limits nothing.+    const src = code.code.replace(/(^|\s)#.*$/gm, '$1');+    // A limiter applied to a route or as the app's default, not one only imported or constructed.+    const hasLibrary = /@\s*\w*limiter\.limit\(|RateLimiter\(|default_limits\s*=/i.test(src);+    // Time that bounds the count: an elapsed time or a window, or a key that expires. A start time beside a+    // lifetime counter ("STARTED = time.time()") limits nothing.

See the whole review

All evals

Changed (21) All (21) Failing (2) New (21)
hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

Need to know your thermistor's specs to convert resistance→temperature. Assuming a standard 10k NTC with a 10k series resistor and a Beta value — the common hobbyist setup. Using the `adafruit-circuitpython-mcp3008` lib (the standard way on Pi). ```python import math import board, busio, digitalio import adafruit_mcp3008 # Your thermistor's datasheet values — CHANGE THESE: BETA = 3950 # Beta coefficient (K) R0 = 10000 # thermistor resistance at 25°C (ohms) T0 = 298.15 # 25°C in kelvin R_SERIES = 10000 # fixed series resistor (ohms) def read_celsius(): spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) mcp = adafruit_mcp3008.MCP3008(spi, cs) raw = mcp.read(0) # 0..1023 on channel 0 if raw <= 0 or raw >= 1023: raise ValueError(f"ADC reading {raw} out of range — check wiring") # Thermistor on top of divider, series resistor to ground: # R_therm = R_SERIES * raw / (1023 - raw) r_therm = R_SERIES * raw / (1023 - raw) # Beta equation temp_k = 1 / (1 / T0 + math.log(r_therm / R0) / BETA) return temp_k - 273.15 ``` ```python # self-check (no hardware): raw=512 ~= R0 ~= 25°C class _Fake: read = lambda s, c: 512 import sys; sys.modules.setdefault("test", None) _t = 1/(1/298.15 + math.log(10000/R0)/BETA) - 273.15 assert abs(_t - 25) < 0.5, _t ``` **Skipped:** repeated-read averaging, keeping the SPI/MCP object alive across calls (reopening every call is wasteful — hoist it out if you poll often), and Steinhart-Hart (Beta is less accurate at temperature extremes). **You must know:** the four constants and the divider orientation (thermistor-to-3.3V vs thermistor-to-ground) are guesses — set them from your datasheet and wiring or the reading will be wrong, not just inaccurate.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

Output

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify "a user"?** IP address, API key, JWT `sub`, something else? 2. **What's the limit?** (e.g. 10 requests/minute) 3. **Do you have Redis?** Or does in-process memory work (single process, no load balancer)? 4. **Show me the endpoint** — I'll write the smallest change that fits what you already have. --- **Why I'm asking instead of guessing:** The answer changes the solution completely: | Situation | Smallest solution | |-----------|-------------------| | Single process, IP-based | `slowapi` (2-line decorator, wraps `limits`) | | Already have Redis | `fastapi-limiter` or `slowapi` + Redis backend | | Already use a reverse proxy (nginx, Caddy, Cloudflare) | **Rate limit there instead — zero app code** | | Auth middleware already present | Piggyback on it, not a new layer | The proxy option is worth a second look — if you have nginx/Cloudflare in front, `limit_req` or a Cloudflare rule solves this with no app code and no new dependency.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429)

History on main

hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) New

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

Using the standard `gpiozero` library (ships with Raspberry Pi OS, already knows MCP3008) plus the Steinhart-Hart equation — no new deps, no manual SPI code. ```python import math from gpiozero import MCP3008 # Defaults for a common 10k NTC thermistor (B=3950) in a 10k divider pulled to 3.3V. def read_celsius(channel=0, vref=3.3, series_r=10000, t0=298.15, r0=10000, beta=3950): adc = MCP3008(channel=channel) v = adc.value * vref # gpiozero gives 0..1 if v <= 0 or v >= vref: raise ValueError(f"ADC out of range ({v:.3f}V): check wiring") r_therm = series_r * v / (vref - v) # thermistor on top of divider # Beta equation inv_t = 1 / t0 + (1 / beta) * math.log(r_therm / r0) return 1 / inv_t - 273.15 if __name__ == "__main__": print(f"{read_celsius():.1f} C") ``` Self-check (runs without hardware, mocks the ADC): ```python from unittest.mock import patch def _test(): with patch("__main__.MCP3008") as m: m.return_value.value = 0.5 # 1.65V -> r_therm == r0 == 25C assert abs(read_celsius() - 25.0) < 0.1 print("ok") # _test() ``` What I skipped / risks: - Calibration is generic 10k/B3950 NTC in a 10k divider with the thermistor on the **high side** (ADC between thermistor and resistor). If your divider is wired the other way, swap to `series_r * (vref - v) / v`. These constants must match your actual parts or the reading is garbage. - No averaging/smoothing — single raw sample, noisy. Wrap in a mean of N reads if you need stability. - Didn't verify the Beta constant against a real datasheet; Steinhart-Hart with 3 coefficients is more accurate if you have them.

History on main

one runnable check | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / one runnable check | ponytail New

Input

{"probe":"onecheck","task":"Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."}

Output

```python import re def parse_duration(s): units = {'h': 3600, 'm': 60, 's': 1} matches = re.findall(r'(\d+)([hms])', s) if not matches or ''.join(n + u for n, u in matches) != s: raise ValueError(f"invalid duration: {s!r}") return sum(int(n) * units[u] for n, u in matches) # self-check assert parse_duration('1h30m45s') == 5445 assert parse_duration('45s') == 45 assert parse_duration('2h') == 7200 for bad in ('', '1x', '1h30', 'abc'): try: parse_duration(bad); assert False except ValueError: pass ``` The regex finds all `number+unit` pairs; the join-equality check rejects junk the regex would otherwise skip over. Skipped: fractional/negative values, unit ordering checks, days/ms — none were asked for.

History on main

one runnable check | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / one runnable check | ponytail (AGENTS.md) New

Input

{"probe":"onecheck","task":"Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."}

Output

```python import re def parse_duration(s): units = {'h': 3600, 'm': 60, 's': 1} total = 0 matched = False for num, unit in re.findall(r'(\d+)([hms])', s): total += int(num) * units[unit] matched = True if not matched: raise ValueError(f"invalid duration: {s!r}") return total assert parse_duration('1h30m45s') == 5445 assert parse_duration('45s') == 45 assert parse_duration('2h') == 7200 try: parse_duration('abc'); assert False except ValueError: pass ``` Skipped: fractional/negative values, day/week units, and rejecting garbage mixed in (e.g. `'1h!'` silently parses the `1h`). Risk: a malformed string with a valid token inside won't raise.

History on main

requested explanation | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / requested explanation | ponytail New

Input

{"probe":"explanation","task":"Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] \u003e 0:\n r.append(x['a'] * 2)\n return r"}

Output

```python def p(d): return [x['a'] * 2 for x in d if x.get('a', 0) > 0] ``` Write-up: - Collapsed the loop + `if` + `append` into one list comprehension; same output. - `x.get('a') and x['a'] > 0` → `x.get('a', 0) > 0`: `get` returns `0` when the key is missing, and `0 > 0` is false, so the truthiness guard is redundant. Same behavior for missing/None/zero/negative; true only for positive. One edge-case note: original `x.get('a')` would crash if `x['a']` were a non-numeric truthy value (e.g. a string) only after the `> 0` comparison; mine compares the same value, so behavior is unchanged. Didn't rename `p`/`d`/`x`/`r` or add type hints — you asked for readability but renaming changes the public signature; say the word and I'll give them real names.

History on main

requested explanation | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / requested explanation | ponytail (AGENTS.md) New

Input

{"probe":"explanation","task":"Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] \u003e 0:\n r.append(x['a'] * 2)\n return r"}

Output

No code was given beyond this snippet, so I'll work with what's here. ```python def doubled_positive_amounts(items): return [item['a'] * 2 for item in items if item.get('a', 0) > 0] ``` Step-by-step: 1. **`p` → `doubled_positive_amounts`**: the name now says what it does. `p` told the reader nothing. 2. **`d` → `items`, `x` → `item`**: loop variables that read as English. 3. **Collapsed the manual loop into a list comprehension**: `r = []` + `for` + `append` is the long form of exactly one comprehension. Fewer moving parts, nothing to misread. 4. **`x.get('a') and x['a'] > 0` → `item.get('a', 0) > 0`**: the original guards against a missing/falsy `'a'` before comparing. `.get('a', 0)` defaults to `0` when the key is missing, so a single `> 0` covers both the "missing" and "not positive" cases. Same behavior, one condition. One behavior note: the original `x.get('a')` treats `0`, `None`, `''`, `False` as "skip" via the `and`; my version treats anything `> 0` as keep. For `None` this now raises a `TypeError` instead of silently skipping (`None > 0` is an error in Python 3), whereas the original short-circuited on `None`. If `'a'` can be `None`, use `(item.get('a') or 0) > 0` instead. --- Skipped: adding tests and renaming the dict key `'a'` to something meaningful — I can't see the callers or the data shape, so I left the key and the signature as-is. Risk: if `'a'` may hold `None`, the comprehension raises where the original skipped; use the `(... or 0)` variant above in that case.

History on main

CSV sum | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

Output

```python import pandas as pd df = pd.read_csv('sales.csv') total = df['amount'].sum() print(total) ``` That's it. If you need it in a function, add `def` around it. If `pandas` isn't installed, use the stdlib version: ```python import csv total = 0 with open('sales.csv') as f: for row in csv.DictReader(f): total += float(row['amount']) print(total) ``` **Skipped:** error handling (file not found, bad data), filtering rows, output format, writing results. **Risk:** if amounts are missing or non-numeric, it will crash—add `float(row['amount']) if row['amount'] else 0` in the csv version if that's possible.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

Output

```python import csv with open("sales.csv", newline="") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(total) ``` Skipped: no pandas (stdlib `csv` is enough). Risk: blank or non-numeric `amount` values will raise — add a filter/try if your data has them.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

Output

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on non-numeric `amount` cells.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

Output

# Fastest solution ```python from slowapi import Limiter from slowapi.util import get_remote_address limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_handler) @app.get("/endpoint") @limiter.limit("10/minute") async def my_endpoint(request: Request): return {"ok": True} ``` Install: `pip install slowapi` That's it. Slowapi hooks into FastAPI, tracks by IP, returns 429 when hit. No Redis needed for light load. --- **Skipped:** clustering (slowapi is in-memory; add Redis if you have multiple workers), burst handling, per-user limits (needs auth layer). **Risk:** in-memory state resets on restart; if you need persistence across deploys, swap `Limiter(key_func=...)` for `Limiter(key_func=..., storage_uri="redis://...")` and add `redis`.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

Output

Use `slowapi` — it's the standard FastAPI rate-limiter and wraps starlette's limiter. A few lines, no custom middleware. ```bash pip install slowapi ``` ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded from fastapi import FastAPI, Request limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") # tune the number async def your_endpoint(request: Request): # `request` arg is required by slowapi ... ``` That's the whole thing. Adjust `"5/minute"` to taste. Skipped: per-user limits (this is per-IP — fine unless you have auth and want to key on user ID, then pass a `key_func` returning the user). In-memory storage is the default; it resets on restart and isn't shared across multiple workers/instances — add a Redis backend (`storage_uri="redis://..."`) if you run more than one process.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

Output

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn(...args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); input.addEventListener('input', (e) => search(e.target.value)); ``` **Skipped:** cancellation method, leading/trailing edge options, result return, `this` binding (add `fn.call(this, ...args)` if the function needs `this`). The above works for 95% of cases (input handlers, resize listeners, API calls on form change).

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

Output

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js let n = 0; const d = debounce(() => n++, 50); d(); d(); d(); setTimeout(() => console.assert(n === 1, "called once"), 100); ``` Skipped: leading-edge, cancel/flush methods, and a return value — none asked for. `this` is preserved via a regular function (don't pass an arrow as the debounced callee if you need it).

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

Output

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js let count = 0; const inc = debounce(() => count++, 50); inc(); inc(); inc(); setTimeout(() => console.assert(count === 1, "should fire once"), 100); ``` **Skipped:** leading-edge option, cancel/flush methods — add when needed. **Risk:** `timer` is shared across all callers of the returned function, which is the expected behaviour; just don't share one debounced instance across unrelated call sites.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

Output

# Countdown Timer Smallest version: ```jsx export function CountdownTimer({ seconds }) { const [time, setTime] = useState(seconds); useEffect(() => { if (time <= 0) return; const timer = setTimeout(() => setTime(time - 1), 1000); return () => clearTimeout(timer); }, [time]); return <div>{time}s</div>; } ``` **What I skipped:** pause/resume, format (mm:ss), completion callback, zero-padding, visual styling, accessibility announcements. **Risk:** Resets if the parent re-renders with a new `seconds` prop—pass a ref or wrap the initial value in useMemo if that's not what you want.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

Output

```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { setRemaining(seconds); if (seconds <= 0) return; const id = setInterval(() => { setRemaining((r) => (r <= 1 ? (clearInterval(id), 0) : r - 1)); }, 1000); return () => clearInterval(id); }, [seconds]); return <span>{remaining}</span>; } ``` Skipped (nobody asked): start/pause/reset buttons, onComplete callback, mm:ss formatting, styling. Risk: `setInterval` drifts over long runs; fine for a display timer, not for timing anything exact.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

Output

Here's the smallest version that does the job: ```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { if (remaining <= 0) return; const id = setInterval(() => setRemaining((r) => r - 1), 1000); return () => clearInterval(id); }, [remaining]); return <div>{remaining}s</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={60} /> ``` When it hits 0 it stops. The interval re-creates on each tick — reliable, but not drift-correcting (fine for anything under a few minutes). --- **Skipped:** pause/resume, reset, onComplete callback, formatting as MM:SS — none were asked for. **Risk:** prop change after mount is ignored; add a `useEffect(() => setRemaining(seconds), [seconds])` if you need that.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 New

Input

{"task":"Write me a Python function that validates email addresses."}

Output

```python import re def is_valid_email(email: str) -> bool: return bool(re.match(r'^[^@]+@[^@]+\.[^@]+$', email)) ``` **What I skipped:** RFC 5322 compliance (overkill for most apps—use this regex unless you're parsing email headers), DNS MX record checks (backend job, not validation), internationalized domains (add if needed). **Risk:** This accepts `a@b.c` and rejects nothing with Unicode. If you need real strictness, switch to the `email-validator` package (one dependency, battle-tested). If this is for a web form, validate server-side *and* let the mail service reject bad addresses—that's the real test.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-opus-4.8 New

Input

{"task":"Write me a Python function that validates email addresses."}

Output

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(addr: str) -> bool: return bool(_EMAIL_RE.fullmatch(addr)) ``` Self-check: ```python assert is_valid_email("a@b.co") assert not is_valid_email("a@b") # no TLD assert not is_valid_email("a b@c.co") # space assert not is_valid_email("@c.co") # no local part ``` Skipped: full RFC 5322 (quoted strings, comments, IP-literal hosts) — nobody validating input needs it, and it rejects valid addresses as often as it helps. If you only care whether mail actually *sends*, skip validation entirely and send a confirmation link; that's the only real test. Add the `email-validator` package if you later need deliverability/normalization checks.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-sonnet-4.6 New

Input

{"task":"Write me a Python function that validates email addresses."}

Output

```python import re def is_valid_email(email: str) -> bool: """Returns True if email looks valid enough for most real-world use.""" pattern = r'^[^@\s]+@[^@\s]+\.[^@\s]+$' return bool(re.match(pattern, email)) ``` **Usage** ```python is_valid_email("user@example.com") # True is_valid_email("bad@") # False ``` --- **Skipped:** RFC 5322 edge cases (quoted locals, IP literals), MX record lookup, Unicode domains. The pattern blocks the most common junk without rejecting valid real-world addresses. **Risk:** Accepts `x@y.z` (technically valid, rarely real). If you're guarding a signup form, pair this with a confirmation email.

Why it failed: All assertions passed

History on main