Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Fix the eval failing on main: hardware calibration #4

gglucass · evalship/failing-1 → main · 145080c (latest is cd02498) · 6 days ago · vs main@c01cd19 · evals run

evals: 20 / 21eval flaws: 0
Ponytail vs ca...14 / 15Ponytail behav...6 / 6+1

Notes

  • ⏳

    Too early to tell: main has run Ponytail vs caveman vs no-skill fewer than 3 times, so evalship can't yet tell these changes from chance.

  • Not comparable: The single-shot configs now score correctness alone, a 0 or 1 (correctness.js:293), where the description says main averaged the raw line count in (loc.js:14). Results show eval_changed 0 and 13 regressed; treat this run as the new baseline for these suites, not main (benchmarks/promptfooconfig.yaml:33).

14 evals got worse in this pull request, too early to tell from chance

Likely cause, in skills/ponytail/SKILL.md

 - Code you move or merge keeps its error handling and validation. - Between options of equal size, take the one that is correct on edge cases. - Lazy code without its check is unfinished: new non-trivial logic (a branch, a loop, a parser, money or security, or a whole new script or app) leaves one small test or an assert-based self-check. Trivial changes need none.+- Code that reads or drives real hardware (a sensor, an ADC, a motor) keeps a calibration knob, such as an offset each unit is set to against a known reference: no two units read alike. - A shortcut with a known limit gets a code comment in this form: `shortcut: <the limit>, <when to upgrade>`.  Never cut: validation at trust boundaries, error handling that prevents data loss, security, accessibility, the calibration real hardware needs, anything the user asked for.

FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5 9.50 → 0.00

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main9.50

# Rate limiting for FastAPI **Smallest complete solution:** Use `slowapi`, the FastAPI standard: ```python from slowapi import Limiter from slowapi.util import get_remote_address limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): return {"data": "value"} ``` **Install:** ```bash pip install slowapi ``` **Add to exception handlers** (one time, in your main app file): ```python from slowapi.errors import RateLimitExceeded from fastapi.responses import JSONResponse @app.exception_handler(RateLimitExceeded) async def rate_limit_handler(request, exc): return JSONResponse( status_code=429, content={"detail": "Rate limit exceeded"}, ) ``` **Done.** Apply `@limiter.limit("N/timeunit")` to any endpoint. Timeunits: `second`, `minute`, `hour`, `day`. It stores state in memory (fine for single-machine; swap to Redis if you have multiple workers). --- **Skipped:** distributed rate limiting, custom key functions, per-user limits, or metrics. **Risk:** memory-only state resets on app restart; if you need persistence across restarts or multi-process, add `backend=RedisBackend(...)` to the Limiter.

✗ This PR0.00

Show me: 1. **The endpoint code** (the function, its job, what data it touches) 2. **The constraint**: requests per second/minute? Per user, IP, or API key? What happens when they exceed it—reject, queue, or rate down? 3. **Your FastAPI setup**: are you running one process, behind a load balancer, or using Redis? One-line preview: if it's one process and you want simple IP-based rate limiting, `slowapi` (already popular in FastAPI apps) is a 3-line add. If you need per-user or multi-process, the answer changes.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429)

History on main

CSV sum | openrouter:anthropic/claude-haiku-4.5

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-haiku-4.5 2.50 → 1.00

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.50

```python import pandas as pd df = pd.read_csv('sales.csv') total = df['amount'].sum() print(total) ``` That's it. If pandas is already a project dependency, use it—it's three lines and handles parsing, missing values, and type conversion. If you need the total as a return value instead of printed, wrap it in a function. **Skipped:** error handling for missing file, type validation, filtering rows. **Risk:** if 'amount' has non-numeric values, `.sum()` will skip them silently (pandas default)—add `.astype(float)` before `.sum()` if that's a problem.

✓ This PR1.00

```python import csv total = sum(float(row['amount']) for row in csv.DictReader(open('sales.csv'))) print(total) ``` **Skipped:** error handling (file not found, bad data), closing the file (context manager would add 2 lines). **Risk:** malformed CSV or missing 'amount' column will crash; if that matters, wrap the sum in try/except and use `with open()`.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-opus-4.8

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-opus-4.8 2.50 → 1.00

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.50

```python import csv with open("sales.csv", newline="") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(total) ``` Skipped: error handling for missing file/bad values (not asked). Risk: assumes a header row named `amount` and that every value parses as a number—one bad cell raises.

✓ This PR1.00

```python import csv with open("sales.csv", newline="") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(total) ``` Skipped: error handling for missing file/column or non-numeric amounts (add try/except if the CSV isn't trusted). Assumes a header row with an `amount` column.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-sonnet-4.6

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-sonnet-4.6 2.00 → 1.00

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on empty `amount` cells — replace `float(...)` with `float(row["amount"] or 0)` if needed.

✓ This PR1.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on blank or non-numeric `amount` cells.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8 5.50 → 1.00

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main5.50

Use `slowapi` — it's the standard FastAPI rate limiter, a few lines. Install: `pip install slowapi` ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") async def your_endpoint(request: Request): # `request` param is required by slowapi ... ``` Change `5/minute` to whatever "spam" means for you. Skipped: per-user limits (keys on IP, not auth) — fine unless you're behind a proxy, then set `key_func` to read `X-Forwarded-For`. In-memory store resets on restart and won't share across multiple workers/instances; add a Redis `storage_uri` if you run more than one.

✓ This PR1.00

What's your setup? It changes the answer: 1. **Single process, one instance** → use `slowapi` (wraps the standard `limits` lib, built for FastAPI). ~5 lines. 2. **Multiple workers/instances** → you need shared state, so `slowapi` + Redis, or do it at your reverse proxy (nginx `limit_req`) and write zero Python. If you have nginx/a load balancer in front, option 2-proxy is the laziest: no code, no deps. Assuming single instance with `slowapi` (most common case): ```python # pip install slowapi from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded from fastapi import Request limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") # tune the number async def your_endpoint(request: Request): # `request` param is required by slowapi ... ``` Two things that bite people: - `get_remote_address` reads the client IP. **Behind a proxy it sees the proxy's IP**, so everyone shares one limit. Fix: run with `--proxy-headers` (uvicorn) and make sure `X-Forwarded-For` is trusted, or key on an API token/user ID instead. - In-memory storage resets on restart and isn't shared across workers. Fine for one process; not fine for `--workers 2+`. Skipped: Redis backend, per-user keying, and I didn't write a test (slowapi is battle-tested; your risk is the IP/proxy config above, not the limiter). Tell me your deployment shape and I'll point you at exactly one of these.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-haiku-4.5

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-haiku-4.5 6.00 → 1.00

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // Only this fires, 300ms later ``` --- **Skipped:** cancellation method, trailing/leading edge options, return value from fn, max-wait limit. Add any if you need them. **Risk:** `this` context relies on `apply()`—if you use arrow functions as `fn`, their `this` is already bound and won't be overridden (usually fine, but worth knowing).

✓ This PR1.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // Only this one fires, 300ms later ``` **What I skipped:** leading/trailing edge options, return value from fn, and a way to cancel. Add them only if you need them—this handles 95% of debounce use cases.

Why it failed: All assertions passed

History on main

8 more under All evals.

0 eval flaws in this pull request

Eval gap in AGENTS.md, dismissed by gglucass as won't fix: checked, all 6 passing hardware replies on #4 set a per-unit offset in code (OFFSET_C = 0.0, offset=0.0); requiring an offset identifier would fail replies that calibrate by measuring their own R0, which the check passes on purpose (tests/behavior.test.js:18, :113).

The hardware probe in behavior.js matches 'calibration knob' and 'known reference' (line 36), the words this rule adds, so a reply that only echoes the rule passes with no per-unit offset. Check passing replies for an actual offset before reading the new passes as the fix.

Untested in hooks/ponytail-instructions.js, dismissed by gglucass as later: the fallback repeats AGENTS.md's rules and runs only when SKILL.md can't be read; a test that it matches AGENTS.md would cover it (hooks/ponytail-instructions.js:44).

hooks/ponytail-instructions.js#getFallbackInstructions has no eval.

All evals

Changed (17) All (21) Failing (1) Now failing (1) Regressed (13) Now passing (2) Improved (1)
CSV sum | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.50

```python import pandas as pd df = pd.read_csv('sales.csv') total = df['amount'].sum() print(total) ``` That's it. If pandas is already a project dependency, use it—it's three lines and handles parsing, missing values, and type conversion. If you need the total as a return value instead of printed, wrap it in a function. **Skipped:** error handling for missing file, type validation, filtering rows. **Risk:** if 'amount' has non-numeric values, `.sum()` will skip them silently (pandas default)—add `.astype(float)` before `.sum()` if that's a problem.

✓ This PR1.00

```python import csv total = sum(float(row['amount']) for row in csv.DictReader(open('sales.csv'))) print(total) ``` **Skipped:** error handling (file not found, bad data), closing the file (context manager would add 2 lines). **Risk:** malformed CSV or missing 'amount' column will crash; if that matters, wrap the sum in try/except and use `with open()`.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.50

```python import csv with open("sales.csv", newline="") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(total) ``` Skipped: error handling for missing file/bad values (not asked). Risk: assumes a header row named `amount` and that every value parses as a number—one bad cell raises.

✓ This PR1.00

```python import csv with open("sales.csv", newline="") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(total) ``` Skipped: error handling for missing file/column or non-numeric amounts (add try/except if the CSV isn't trusted). Assumes a header row with an `amount` column.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on empty `amount` cells — replace `float(...)` with `float(row["amount"] or 0)` if needed.

✓ This PR1.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on blank or non-numeric `amount` cells.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main5.50

Use `slowapi` — it's the standard FastAPI rate limiter, a few lines. Install: `pip install slowapi` ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") async def your_endpoint(request: Request): # `request` param is required by slowapi ... ``` Change `5/minute` to whatever "spam" means for you. Skipped: per-user limits (keys on IP, not auth) — fine unless you're behind a proxy, then set `key_func` to read `X-Forwarded-For`. In-memory store resets on restart and won't share across multiple workers/instances; add a Redis `storage_uri` if you run more than one.

✓ This PR1.00

What's your setup? It changes the answer: 1. **Single process, one instance** → use `slowapi` (wraps the standard `limits` lib, built for FastAPI). ~5 lines. 2. **Multiple workers/instances** → you need shared state, so `slowapi` + Redis, or do it at your reverse proxy (nginx `limit_req`) and write zero Python. If you have nginx/a load balancer in front, option 2-proxy is the laziest: no code, no deps. Assuming single instance with `slowapi` (most common case): ```python # pip install slowapi from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded from fastapi import Request limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") # tune the number async def your_endpoint(request: Request): # `request` param is required by slowapi ... ``` Two things that bite people: - `get_remote_address` reads the client IP. **Behind a proxy it sees the proxy's IP**, so everyone shares one limit. Fix: run with `--proxy-headers` (uvicorn) and make sure `X-Forwarded-For` is trusted, or key on an API token/user ID instead. - In-memory storage resets on restart and isn't shared across workers. Fine for one process; not fine for `--workers 2+`. Skipped: Redis backend, per-user keying, and I didn't write a test (slowapi is battle-tested; your risk is the IP/proxy config above, not the limiter). Tell me your deployment shape and I'll point you at exactly one of these.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // Only this fires, 300ms later ``` --- **Skipped:** cancellation method, trailing/leading edge options, return value from fn, max-wait limit. Add any if you need them. **Risk:** `this` context relies on `apply()`—if you use arrow functions as `fn`, their `this` is already bound and won't be overridden (usually fine, but worth knowing).

✓ This PR1.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // Only this one fires, 300ms later ``` **What I skipped:** leading/trailing edge options, return value from fn, and a way to cancel. Add them only if you need them—this handles 95% of debounce use cases.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js let calls = 0; const d = debounce(() => calls++, 50); d(); d(); d(); setTimeout(() => console.assert(calls === 1, `expected 1, got ${calls}`), 100); ``` Skipped: leading-edge, cancel/flush methods, and a return value (debounced calls are async, so there's nothing to return). Say the word if you need any of those.

✓ This PR1.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` `this` is forwarded via `apply` so it works on methods too. Skipped: cancel/flush methods and leading-edge options — none asked for. No return value from the debounced call (debounced fns run later, so there's nothing to return synchronously).

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Self-check** (runs once in Node or a browser console): ```js // quick smoke-test — remove in production let calls = 0; const inc = debounce(() => calls++, 50); inc(); inc(); inc(); setTimeout(() => console.assert(calls === 1, "debounce failed"), 100); ``` **Skipped:** leading-edge ("immediate") option, cancel/flush methods — not asked for. **Risk:** `this` binding relies on normal `function` syntax; arrow-function callers won't need it, but class method callers will — works correctly either way.

✓ This PR1.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Self-check** (paste into a console or Node): ```js let count = 0; const inc = debounce(() => count++, 50); inc(); inc(); inc(); setTimeout(() => console.assert(count === 1, "should fire once"), 100); ``` --- **Skipped:** leading-edge option, cancel/flush methods, maxWait — add if needed. **Risk:** `timer` is captured in closure, so two callers sharing one debounced function will stomp each other's timer (expected behavior; just worth knowing).

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main7.00

# Countdown Timer ```jsx export function CountdownTimer({ seconds, onComplete }) { const [timeLeft, setTimeLeft] = useState(seconds); useEffect(() => { if (timeLeft <= 0) { onComplete?.(); return; } const timer = setInterval(() => setTimeLeft(t => t - 1), 1000); return () => clearInterval(timer); }, [timeLeft, onComplete]); return <div>{timeLeft}s</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={60} onComplete={() => console.log('done')} /> ``` --- **Skipped:** reset button, pause, display format (mm:ss), animations, styling, aria labels, effect cleanup on unmount edge case. **Risk:** `onComplete` in the dependency array could fire multiple times if you pass a new function every render—wrap it in `useCallback` if that's your pattern. If `seconds` changes mid-countdown, it restarts.

✓ This PR1.00

# Countdown Timer ```jsx export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = React.useState(seconds); React.useEffect(() => { if (remaining <= 0) return; const id = setInterval(() => setRemaining(r => r - 1), 1000); return () => clearInterval(id); }, [remaining]); return <div>{remaining}s</div>; } ``` **Use it:** ```jsx <CountdownTimer seconds={10} /> ``` **Skipped:** pause/resume, visual progress bar, callback on zero, custom formatting, accessibility announcements. **Risk:** if `seconds` prop changes, the timer restarts (usually desired, but flag it if you need smooth handoff).

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main7.00

```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { setRemaining(seconds); if (seconds <= 0) return; const id = setInterval(() => { setRemaining((r) => (r <= 1 ? (clearInterval(id), 0) : r - 1)); }, 1000); return () => clearInterval(id); }, [seconds]); return <span>{remaining}</span>; } ``` Skipped (nobody asked): pause/reset, mm:ss formatting, onComplete callback — add if you need them. Risk: `setInterval` drifts over long runs; fine for a display timer, not for anything billed by the second.

✓ This PR1.00

```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { setRemaining(seconds); if (seconds <= 0) return; const id = setInterval(() => { setRemaining((s) => (s <= 1 ? (clearInterval(id), 0) : s - 1)); }, 1000); return () => clearInterval(id); }, [seconds]); return <span>{remaining}</span>; } ``` Skipped (nobody asked): start/pause/reset controls, mm:ss formatting, onComplete callback, styling. Say the word if you want any. Risk: `setInterval` drifts over long runs and pauses when the tab is backgrounded — fine for a short display timer, not for anything you'd bill against.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main13.50

Here's the minimal component: ```jsx // CountdownTimer.jsx import { useState, useEffect, useRef } from "react"; export default function CountdownTimer({ seconds, onComplete }) { const [remaining, setRemaining] = useState(seconds); const intervalRef = useRef(null); useEffect(() => { if (remaining <= 0) { onComplete?.(); return; } intervalRef.current = setInterval(() => { setRemaining((r) => { if (r <= 1) { clearInterval(intervalRef.current); onComplete?.(); return 0; } return r - 1; }); }, 1000); return () => clearInterval(intervalRef.current); }, []); // run once on mount const m = Math.floor(remaining / 60).toString().padStart(2, "0"); const s = (remaining % 60).toString().padStart(2, "0"); return <div aria-live="polite">{m}:{s}</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={90} onComplete={() => alert("Done!")} /> ``` --- **Skipped:** pause/resume/reset controls, resetting when `seconds` prop changes after mount, and any visual styling — add those if you need them. **Risk:** the interval drifts slightly over long durations (~1 s per few minutes); use `Date`-based logic if precision matters.

✓ This PR1.00

Here's the minimal component: ```jsx // CountdownTimer.jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds, onComplete }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { if (remaining <= 0) { onComplete?.(); return; } const id = setTimeout(() => setRemaining((r) => r - 1), 1000); return () => clearTimeout(id); }, [remaining]); return <span>{remaining}s</span>; } ``` **Usage:** ```jsx <CountdownTimer seconds={30} onComplete={() => alert("Done!")} /> ``` --- **Skipped:** pause/resume, reset, display formatting (mm:ss), accessibility live region, persisting across remounts. **Risk:** If `seconds` prop changes after mount the timer ignores it — add a `useEffect` on `seconds` to reset `remaining` if you need that.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main2.00

```python import re def is_valid_email(email: str) -> bool: return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (full spec is huge and impractical). This catches ~99% of real addresses. If you need strict validation, use `email_validator` package. If you need to confirm the address exists, you must send a confirmation email—no regex does that. **Risk:** This accepts `a@b.c` (technically valid but rare). It rejects nothing malicious. Use this for UX feedback only; validate on the backend before trusting user data.

✓ This PR1.00

```python import re def is_valid_email(email: str) -> bool: return bool(re.match(r'^[^@]+@[^@]+\.[^@]+$', email)) ``` **What I skipped:** RFC 5322 full compliance (250+ lines, breaks on real-world addresses), DNS checks, sending a confirmation link. The regex catches 99% of typos and is what most projects actually need. **Risk:** A few edge cases like `a@b.c` pass (arguably correct), and obfuscated attacks like `a@b.co.uk` pass (as they should). If you must reject something specific, tell me the rule.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main4.50

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.fullmatch(email)) ``` Quick self-check: ```python assert is_valid_email("a@b.co") assert not is_valid_email("a@b") # no TLD assert not is_valid_email("a b@c.co") # space assert not is_valid_email("a@@b.co") # double @ ``` Skipped: full RFC 5322 (quoted locals, IP literals, comments) — nobody needs it and the regex gets monstrous. Risk: this is a format check, not a deliverability check; the only way to know an address works is to send to it.

✓ This PR1.00

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.fullmatch(email)) ``` ```python assert is_valid_email("a@b.co") assert not is_valid_email("a@b") assert not is_valid_email("a b@c.com") assert not is_valid_email("") ``` Skipped: full RFC 5322 grammar, quoted locals, IP-literal domains — nobody asked and that regex is a monster. Risk: this accepts some syntactically-weird-but-rare addresses and rejects nothing real; the only way to truly know an address is valid is to send mail to it. If you need stricter, install `email-validator` instead of growing this.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main3.50

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.fullmatch(email)) ``` **Usage** ```python is_valid_email("user@example.com") # True is_valid_email("bad@") # False ``` --- Skipped: RFC 5322 edge cases (quoted locals, IP literals, comments) — real-world emails don't use them. Risk: rejects technically valid but exotic addresses like `"user name"@example.com`; use a library like [`email-validator`](https://pypi.org/project/email-validator/) if you need strict RFC compliance or MX-record checking.

✓ This PR1.00

```python import re def is_valid_email(email: str) -> bool: """Returns True if email looks valid enough for form validation.""" pattern = r'^[^@\s]+@[^@\s]+\.[^@\s]+$' return bool(re.match(pattern, email)) ``` **Usage:** ```python is_valid_email("user@example.com") # True is_valid_email("bad@") # False ``` --- **Skipped:** RFC 5322 edge cases (quoted locals, IP literals, comments) — they're vanishingly rare and most mail servers reject them anyway. This matches what every major site actually validates. **Risk:** Accepts `x@y.z` (technically valid, practically junk) — add a TLD-length check or a DNS lookup if you need stronger guarantees.

Why it failed: All assertions passed

History on main