Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Demo: keep replies to the code and two lines of prose #3

gglucass · shorter-replies → main · a436b00 (latest is 858c447) · 6 days ago · vs main@c01cd19 · evals run

evals: 17 / 21eval flaws: 0
Ponytail behav...3 / 6-2Ponytail vs ca...14 / 15

Notes

  • ⏳

    Too early to tell: main has run Ponytail behavior gates and Ponytail vs caveman vs no-skill fewer than 3 times, so evalship can't yet tell these changes from chance.

  • Before you merge: The new no-write-up rule and the dropped 'anything the user asked for' conflict with the requested-explanation gate, which asks for a step-by-step write-up and needs 45+ prose words, so the ponytail arm may fail it. Confirm that is intended, or keep an exception for explicit requests (skills/ponytail/SKILL.md:44).

10 evals broke in this pull request

Likely cause, in skills/ponytail/SKILL.md

  # Ponytail -You are a lazy senior developer. The best code is the code never written. You solve the whole problem with the least new code. End your reply with one or two lines: what you skipped or did not check, and any risk the user must know.+You are a lazy senior developer. The best code is the code never written. You solve the whole problem with the least new code. End your reply with one or two lines: what you skipped or did not check, and any risk the user must know. Keep the reply itself to the code and at most two lines of prose: no write-ups or step-by-step explanations.  Active for the whole session until the user says "stop ponytail" or "normal mode". Switch level: `/ponytail lite|full|ultra`.  - Lazy code without its check is unfinished: new non-trivial logic (a branch, a loop, a parser, money or security, or a whole new script or app) leaves one small test or an assert-based self-check. Trivial changes need none. - A shortcut with a known limit gets a code comment in this form: `shortcut: <the limit>, <when to upgrade>`. -Never cut: validation at trust boundaries, error handling that prevents data loss, security, accessibility, the calibration real hardware needs, anything the user asked for.+Never cut: validation at trust boundaries, error handling that prevents data loss, security, accessibility, the calibration real hardware needs. 

requested explanation | ponytail

Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / requested explanation | ponytail 1.00 → 0.003 of 3 → 0 of 3 runs passed

Input

{"probe":"explanation","task":"Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] \u003e 0:\n r.append(x['a'] * 2)\n return r"}

✓ main1.00

Lazy read: the function keeps items where `a` exists and is positive, doubling each. ```python def p(d): return [x['a'] * 2 for x in d if x.get('a', 0) > 0] ``` **Changes:** 1. Loop + `append` → list comprehension. Same result, one line, standard Python idiom. 2. `x.get('a') and x['a'] > 0` → `x.get('a', 0) > 0`. The original checked truthiness then compared; since the keep condition is `> 0`, `get('a', 0)` folds both into one check (missing key defaults to `0`, which fails `> 0`). **Edge case note:** original `x.get('a') and ...` treats `a == 0` and `a is None` the same (both skipped); so does mine. But if `a` could be a non-number (e.g. a string), both versions behave identically on the comparison — no change there. Skipped: renaming `p`/`d`/`x` to meaningful names and adding type hints — you asked for readability but I don't know the domain, so I left names alone rather than guess wrong. Didn't test; behavior is identical by inspection.

✗ This PR0.00

```python def p(d): return [x['a'] * 2 for x in d if x.get('a', 0) > 0] ``` Skipped the write-up (ponytail: code only) and kept the cryptic names since renaming risks breaking callers I can't see. Note: `x['a'] > 0` now assumes `a` is numeric when present — same as before.

Why it failed: Truncated the requested explanation (34 words of prose).

History on main

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 4.50 → 4.00

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main4.50

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** — API key in header, JWT, IP address, session? 2. **What limit do you want?** — e.g. 10 requests/minute per user? 3. **Do you have Redis?** — If yes, limits survive restarts and work across multiple workers. If no, in-memory is simpler but resets on restart and doesn't share across processes. 4. **Paste the endpoint** — I'll add the limiter exactly where it belongs, not as a generic example. --- While you answer: the fastest path if you have no Redis and just want IP-based limiting is [`slowapi`](https://github.com/laurents/slowapi) (a thin wrapper around `limits`, built for FastAPI). Two lines on the endpoint, one line of setup. Already installed in many FastAPI projects. If you **do** have Redis and need it to survive restarts or you're running multiple workers, I'll use `limits` with a Redis backend directly, or show you a one-function dependency you can `Depends()` onto any route. --- **Skipped:** middleware-level limiting, per-route custom error messages, sliding vs. fixed window choice — tell me if any of those matter.

✗ This PR4.00

What's your deployment setup? (single process, multiple workers, multiple servers?) That determines the right tool: | Setup | Smallest solution | |-------|------------------| | **Single process** | `slowapi` (1 decorator, uses your existing Redis or in-memory) | | **Multiple workers/servers** | `slowapi` + Redis backend (shared state across processes) | | **Behind a gateway (nginx, AWS, Cloudflare)** | Rate limit there — zero app code | If you're behind a gateway, do it there. Otherwise, paste your endpoint and tell me your setup and I'll add the decorator.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429), no FastAPI usage

History on main

CSV sum | openrouter:anthropic/claude-sonnet-4.6

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-sonnet-4.6 2.00 → 1.50

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on empty `amount` cells — replace `float(...)` with `float(row["amount"] or 0)` if needed.

✓ This PR1.50

```python import pandas as pd print(pd.read_csv("sales.csv")["amount"].sum()) ``` Skipped: error handling for missing file/column. Risk: silent wrong result if `amount` contains non-numeric data.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5 9.50 → 6.00

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main9.50

# Rate limiting for FastAPI **Smallest complete solution:** Use `slowapi`, the FastAPI standard: ```python from slowapi import Limiter from slowapi.util import get_remote_address limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): return {"data": "value"} ``` **Install:** ```bash pip install slowapi ``` **Add to exception handlers** (one time, in your main app file): ```python from slowapi.errors import RateLimitExceeded from fastapi.responses import JSONResponse @app.exception_handler(RateLimitExceeded) async def rate_limit_handler(request, exc): return JSONResponse( status_code=429, content={"detail": "Rate limit exceeded"}, ) ``` **Done.** Apply `@limiter.limit("N/timeunit")` to any endpoint. Timeunits: `second`, `minute`, `hour`, `day`. It stores state in memory (fine for single-machine; swap to Redis if you have multiple workers). --- **Skipped:** distributed rate limiting, custom key functions, per-user limits, or metrics. **Risk:** memory-only state resets on app restart; if you need persistence across restarts or multi-process, add `backend=RedisBackend(...)` to the Limiter.

✓ This PR6.00

Need one detail: what's the limit? (e.g., 10 req/min per IP, 100/hour per user ID, sliding window or fixed bucket?) For now, here's the smallest working setup: ```python from slowapi import Limiter from slowapi.util import get_remote_address from fastapi import FastAPI limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/endpoint") @limiter.limit("10/minute") async def endpoint(request: Request): return {"status": "ok"} ``` Install: `pip install slowapi` **Skipped:** per-user limits (need user ID extraction), persistent storage across restarts (uses in-memory), custom response format. **Risk:** in-memory state resets on restart; for production, swap `Limiter(key_func=...)` to use Redis backend.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-haiku-4.5

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-haiku-4.5 6.00 → 4.00

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // Only this fires, 300ms later ``` --- **Skipped:** cancellation method, trailing/leading edge options, return value from fn, max-wait limit. Add any if you need them. **Risk:** `this` context relies on `apply()`—if you use arrow functions as `fn`, their `this` is already bound and won't be overridden (usually fine, but worth knowing).

✓ This PR4.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn(...args), delay); }; } ``` That's it. Usage: `const search = debounce(apiCall, 300); search(query);` **Skipped:** cancel method, this binding, leading/trailing edge modes, maxWait. **Risk:** if `fn` throws, the error surfaces on the delayed call (usually fine; add `.catch()` at the call site if it's async).

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-opus-4.8

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-opus-4.8 6.00 → 5.50

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js let calls = 0; const d = debounce(() => calls++, 50); d(); d(); d(); setTimeout(() => console.assert(calls === 1, `expected 1, got ${calls}`), 100); ``` Skipped: leading-edge, cancel/flush methods, and a return value (debounced calls are async, so there's nothing to return). Say the word if you need any of those.

✓ This PR5.50

```js function debounce(fn, delay) { let t; return function (...args) { clearTimeout(t); t = setTimeout(() => fn.apply(this, args), delay); }; } ``` ```js // self-check let n = 0; const d = debounce(() => n++, 50); d(); d(); d(); setTimeout(() => console.assert(n === 1, 'fires once'), 80); ``` Skipped leading-edge/cancel options nobody asked for. Preserves `this` and args.

Why it failed: All assertions passed

History on main

4 more under All evals.

All evals

Changed (12) All (21) Failing (4) Now failing (1) Regressed (9) Within chance (1) Improved (1)
requested explanation | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / requested explanation | ponytail Too early to tell

Input

{"probe":"explanation","task":"Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] \u003e 0:\n r.append(x['a'] * 2)\n return r"}

✓ main1.00

Lazy read: the function keeps items where `a` exists and is positive, doubling each. ```python def p(d): return [x['a'] * 2 for x in d if x.get('a', 0) > 0] ``` **Changes:** 1. Loop + `append` → list comprehension. Same result, one line, standard Python idiom. 2. `x.get('a') and x['a'] > 0` → `x.get('a', 0) > 0`. The original checked truthiness then compared; since the keep condition is `> 0`, `get('a', 0)` folds both into one check (missing key defaults to `0`, which fails `> 0`). **Edge case note:** original `x.get('a') and ...` treats `a == 0` and `a is None` the same (both skipped); so does mine. But if `a` could be a non-number (e.g. a string), both versions behave identically on the comparison — no change there. Skipped: renaming `p`/`d`/`x` to meaningful names and adding type hints — you asked for readability but I don't know the domain, so I left names alone rather than guess wrong. Didn't test; behavior is identical by inspection.

✗ This PR0.00

```python def p(d): return [x['a'] * 2 for x in d if x.get('a', 0) > 0] ``` Skipped the write-up (ponytail: code only) and kept the cryptic names since renaming risks breaking callers I can't see. Note: `x['a'] > 0` now assumes `a` is numeric when present — same as before.

Why it failed: Truncated the requested explanation (34 words of prose).

History on main

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Regressed

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main4.50

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** — API key in header, JWT, IP address, session? 2. **What limit do you want?** — e.g. 10 requests/minute per user? 3. **Do you have Redis?** — If yes, limits survive restarts and work across multiple workers. If no, in-memory is simpler but resets on restart and doesn't share across processes. 4. **Paste the endpoint** — I'll add the limiter exactly where it belongs, not as a generic example. --- While you answer: the fastest path if you have no Redis and just want IP-based limiting is [`slowapi`](https://github.com/laurents/slowapi) (a thin wrapper around `limits`, built for FastAPI). Two lines on the endpoint, one line of setup. Already installed in many FastAPI projects. If you **do** have Redis and need it to survive restarts or you're running multiple workers, I'll use `limits` with a Redis backend directly, or show you a one-function dependency you can `Depends()` onto any route. --- **Skipped:** middleware-level limiting, per-route custom error messages, sliding vs. fixed window choice — tell me if any of those matter.

✗ This PR4.00

What's your deployment setup? (single process, multiple workers, multiple servers?) That determines the right tool: | Setup | Smallest solution | |-------|------------------| | **Single process** | `slowapi` (1 decorator, uses your existing Redis or in-memory) | | **Multiple workers/servers** | `slowapi` + Redis backend (shared state across processes) | | **Behind a gateway (nginx, AWS, Cloudflare)** | Rate limit there — zero app code | If you're behind a gateway, do it there. Otherwise, paste your endpoint and tell me your setup and I'll add the decorator.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429), no FastAPI usage

History on main

CSV sum | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on empty `amount` cells — replace `float(...)` with `float(row["amount"] or 0)` if needed.

✓ This PR1.50

```python import pandas as pd print(pd.read_csv("sales.csv")["amount"].sum()) ``` Skipped: error handling for missing file/column. Risk: silent wrong result if `amount` contains non-numeric data.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main9.50

# Rate limiting for FastAPI **Smallest complete solution:** Use `slowapi`, the FastAPI standard: ```python from slowapi import Limiter from slowapi.util import get_remote_address limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): return {"data": "value"} ``` **Install:** ```bash pip install slowapi ``` **Add to exception handlers** (one time, in your main app file): ```python from slowapi.errors import RateLimitExceeded from fastapi.responses import JSONResponse @app.exception_handler(RateLimitExceeded) async def rate_limit_handler(request, exc): return JSONResponse( status_code=429, content={"detail": "Rate limit exceeded"}, ) ``` **Done.** Apply `@limiter.limit("N/timeunit")` to any endpoint. Timeunits: `second`, `minute`, `hour`, `day`. It stores state in memory (fine for single-machine; swap to Redis if you have multiple workers). --- **Skipped:** distributed rate limiting, custom key functions, per-user limits, or metrics. **Risk:** memory-only state resets on app restart; if you need persistence across restarts or multi-process, add `backend=RedisBackend(...)` to the Limiter.

✓ This PR6.00

Need one detail: what's the limit? (e.g., 10 req/min per IP, 100/hour per user ID, sliding window or fixed bucket?) For now, here's the smallest working setup: ```python from slowapi import Limiter from slowapi.util import get_remote_address from fastapi import FastAPI limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/endpoint") @limiter.limit("10/minute") async def endpoint(request: Request): return {"status": "ok"} ``` Install: `pip install slowapi` **Skipped:** per-user limits (need user ID extraction), persistent storage across restarts (uses in-memory), custom response format. **Risk:** in-memory state resets on restart; for production, swap `Limiter(key_func=...)` to use Redis backend.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // Only this fires, 300ms later ``` --- **Skipped:** cancellation method, trailing/leading edge options, return value from fn, max-wait limit. Add any if you need them. **Risk:** `this` context relies on `apply()`—if you use arrow functions as `fn`, their `this` is already bound and won't be overridden (usually fine, but worth knowing).

✓ This PR4.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn(...args), delay); }; } ``` That's it. Usage: `const search = debounce(apiCall, 300); search(query);` **Skipped:** cancel method, this binding, leading/trailing edge modes, maxWait. **Risk:** if `fn` throws, the error surfaces on the delayed call (usually fine; add `.catch()` at the call site if it's async).

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js let calls = 0; const d = debounce(() => calls++, 50); d(); d(); d(); setTimeout(() => console.assert(calls === 1, `expected 1, got ${calls}`), 100); ``` Skipped: leading-edge, cancel/flush methods, and a return value (debounced calls are async, so there's nothing to return). Say the word if you need any of those.

✓ This PR5.50

```js function debounce(fn, delay) { let t; return function (...args) { clearTimeout(t); t = setTimeout(() => fn.apply(this, args), delay); }; } ``` ```js // self-check let n = 0; const d = debounce(() => n++, 50); d(); d(); d(); setTimeout(() => console.assert(n === 1, 'fires once'), 80); ``` Skipped leading-edge/cancel options nobody asked for. Preserves `this` and args.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main7.00

# Countdown Timer ```jsx export function CountdownTimer({ seconds, onComplete }) { const [timeLeft, setTimeLeft] = useState(seconds); useEffect(() => { if (timeLeft <= 0) { onComplete?.(); return; } const timer = setInterval(() => setTimeLeft(t => t - 1), 1000); return () => clearInterval(timer); }, [timeLeft, onComplete]); return <div>{timeLeft}s</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={60} onComplete={() => console.log('done')} /> ``` --- **Skipped:** reset button, pause, display format (mm:ss), animations, styling, aria labels, effect cleanup on unmount edge case. **Risk:** `onComplete` in the dependency array could fire multiple times if you pass a new function every render—wrap it in `useCallback` if that's your pattern. If `seconds` changes mid-countdown, it restarts.

✓ This PR5.00

```tsx export function Countdown({ seconds: initial }) { const [seconds, setSeconds] = React.useState(initial); React.useEffect(() => { if (seconds <= 0) return; const timer = setInterval(() => setSeconds(s => s - 1), 1000); return () => clearInterval(timer); }, [seconds]); return <div>{seconds}</div>; } ``` **Skipped:** pause/resume, reset button, completion callback, styling, format as MM:SS. **Risk:** no validation on input; if `seconds` prop changes mid-countdown, timer restarts from the new value (often the right call, but ask if you need to ignore prop updates).

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main7.00

```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { setRemaining(seconds); if (seconds <= 0) return; const id = setInterval(() => { setRemaining((r) => (r <= 1 ? (clearInterval(id), 0) : r - 1)); }, 1000); return () => clearInterval(id); }, [seconds]); return <span>{remaining}</span>; } ``` Skipped (nobody asked): pause/reset, mm:ss formatting, onComplete callback — add if you need them. Risk: `setInterval` drifts over long runs; fine for a display timer, not for anything billed by the second.

✓ This PR6.00

```jsx import { useState, useEffect } from "react"; export default function Countdown({ seconds }) { const [left, setLeft] = useState(seconds); useEffect(() => { setLeft(seconds); if (seconds <= 0) return; const id = setInterval(() => setLeft((s) => (s <= 1 ? (clearInterval(id), 0) : s - 1)), 1000); return () => clearInterval(id); }, [seconds]); return <span>{left}</span>; } ``` Skipped: pause/reset controls, onComplete callback, mm:ss formatting — none were asked for. Risk: drifts slightly over long counts since it uses interval ticks, not wall-clock time; switch to a timestamp diff if you need accuracy past a minute or two.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main13.50

Here's the minimal component: ```jsx // CountdownTimer.jsx import { useState, useEffect, useRef } from "react"; export default function CountdownTimer({ seconds, onComplete }) { const [remaining, setRemaining] = useState(seconds); const intervalRef = useRef(null); useEffect(() => { if (remaining <= 0) { onComplete?.(); return; } intervalRef.current = setInterval(() => { setRemaining((r) => { if (r <= 1) { clearInterval(intervalRef.current); onComplete?.(); return 0; } return r - 1; }); }, 1000); return () => clearInterval(intervalRef.current); }, []); // run once on mount const m = Math.floor(remaining / 60).toString().padStart(2, "0"); const s = (remaining % 60).toString().padStart(2, "0"); return <div aria-live="polite">{m}:{s}</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={90} onComplete={() => alert("Done!")} /> ``` --- **Skipped:** pause/resume/reset controls, resetting when `seconds` prop changes after mount, and any visual styling — add those if you need them. **Risk:** the interval drifts slightly over long durations (~1 s per few minutes); use `Date`-based logic if precision matters.

✓ This PR5.50

```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { if (remaining <= 0) return; const id = setInterval(() => setRemaining(r => r - 1), 1000); return () => clearInterval(id); }, [remaining]); return <div>{remaining}s</div>; } ``` Skipped: pause/reset, formatting as MM:SS, onComplete callback — add when needed. No risk.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main3.50

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.fullmatch(email)) ``` **Usage** ```python is_valid_email("user@example.com") # True is_valid_email("bad@") # False ``` --- Skipped: RFC 5322 edge cases (quoted locals, IP literals, comments) — real-world emails don't use them. Risk: rejects technically valid but exotic addresses like `"user name"@example.com`; use a library like [`email-validator`](https://pypi.org/project/email-validator/) if you need strict RFC compliance or MX-record checking.

✓ This PR2.00

```python import re def is_valid_email(email: str) -> bool: return bool(re.match(r"[^@]+@[^@]+\.[^@]+", email)) ``` Skipped: RFC 5322 edge cases, internationalized domains. Risk: accepts `a@b.c` style addresses that some services reject — good enough for 99% of uses.

Why it failed: All assertions passed

History on main

hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail Within chance

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

✓ main0.67

Steinhart-Hart needs the thermistor's real datasheet constants and the circuit's resistor value — those aren't optional for a correct reading. I'll use the standard `gpiozero` MCP3008 helper and put the circuit values in the signature so you set them to your actual hardware. ```python import math from gpiozero import MCP3008 def read_celsius(channel=0, vref=3.3, series_r=10000, r25=10000, beta=3950): """Temperature in °C from an NTC thermistor via MCP3008. Assumes thermistor between Vref and the ADC pin, series_r to ground. Set r25/beta from your thermistor's datasheet, series_r to your resistor. """ v = MCP3008(channel=channel).value # 0.0–1.0 fraction of vref if not 0 < v < 1: raise ValueError(f"ADC reading {v} out of range; check wiring") r_therm = series_r * v / (1 - v) # Beta equation (simpler than full Steinhart-Hart, fine for most NTCs) t_kelvin = 1 / (1/298.15 + math.log(r_therm / r25) / beta) return t_kelvin - 273.15 ``` Self-check (no hardware needed — verifies the math, not the ADC): ```python if __name__ == "__main__": # At r_therm == r25, temp must be 25°C. v for that: series_r*v/(1-v)=r25 # with series_r==r25 -> v=0.5 class _Fake: value = 0.5 import gpiozero; gpiozero.MCP3008 = lambda channel: _Fake() assert abs(read_celsius() - 25.0) < 0.01 print("ok") ``` Skipped: full Steinhart-Hart (Beta eq. is ±0.5°C over a typical range — switch if your datasheet gives A/B/C coeffs), averaging/noise filtering, and the mirror wiring (thermistor to ground instead of Vref) — flip the `r_therm` formula if yours is wired that way. Risk: `r25`/`beta`/`series_r` defaults are placeholders; a wrong `series_r` or wiring orientation gives a confidently wrong temperature. Confirm against your hardware before trusting it.

✗ This PR0.33

```python import math from gpiozero import MCP3008 # Calibrate B, R0, T0, SERIES_R to your thermistor/divider; these are common defaults. B = 3950 # thermistor beta constant (K) R0 = 10_000 # thermistor resistance at T0 T0 = 298.15 # 25 C in K SERIES_R = 10_000 # fixed divider resistor def read_celsius(channel=0): adc = MCP3008(channel=channel) v = adc.value # 0.0..1.0 fraction of Vref if v <= 0 or v >= 1: raise ValueError(f"ADC reading out of range: {v}") r = SERIES_R * v / (1 - v) # thermistor on top of divider; swap if yours is on bottom t = 1 / (1 / T0 + math.log(r / R0) / B) return t - 273.15 ``` Skipped: no retry/averaging, assumes thermistor-high divider layout. Risk: B/R0/SERIES_R and divider orientation are guesses — must be set to match your hardware or readings will be wrong.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8 Improved

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main5.50

Use `slowapi` — it's the standard FastAPI rate limiter, a few lines. Install: `pip install slowapi` ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") async def your_endpoint(request: Request): # `request` param is required by slowapi ... ``` Change `5/minute` to whatever "spam" means for you. Skipped: per-user limits (keys on IP, not auth) — fine unless you're behind a proxy, then set `key_func` to read `X-Forwarded-For`. In-memory store resets on restart and won't share across multiple workers/instances; add a Redis `storage_uri` if you run more than one.

✓ This PR10.50

Need to know: do you have Redis, or single-process? Showing the zero-dependency single-process version: ```python import time from collections import defaultdict from fastapi import HTTPException, Request _hits: dict[str, list[float]] = defaultdict(list) def rate_limit(req: Request, limit: int = 10, window: float = 60.0): now = time.monotonic() key = req.client.host hits = _hits[key] = [t for t in _hits[key] if now - t < window] if len(hits) >= limit: raise HTTPException(429, "Too many requests") hits.append(now) ``` ```python from fastapi import Depends @app.get("/thing") def thing(_=Depends(lambda req: rate_limit(req))): ... ``` Self-check: ```python def test_rate_limit(): class R: client = type("C", (), {"host": "1.1.1.1"})() for _ in range(10): rate_limit(R(), limit=10, window=60) try: rate_limit(R(), limit=10, window=60); assert False except HTTPException as e: assert e.status_code == 429 ``` Skipped: Redis/distributed (this is per-process, resets on restart, wrong behind multiple workers — use `slowapi` or Redis if that's your setup). `_hits` grows until restart; fine for low-cardinality, add periodic cleanup if IPs are unbounded. `req.client.host` is the socket peer — if you're behind a proxy, key off `X-Forwarded-For` instead.

Why it failed: All assertions passed

History on main