Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Commit 7e38f14 on main

6 days ago · vs main@c01cd19 · evals run

evals: 17 / 21
Ponytail behav...4 / 6-1Ponytail vs ca...13 / 15-1

Notes

  • ⏳

    Too early to tell: main has run Ponytail vs caveman vs no-skill fewer than 3 times, so evalship can't yet tell these changes from chance.

2 evals fail with this pull request's eval changes

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 4.50 → 0.00

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main4.50

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** — API key in header, JWT, IP address, session? 2. **What limit do you want?** — e.g. 10 requests/minute per user? 3. **Do you have Redis?** — If yes, limits survive restarts and work across multiple workers. If no, in-memory is simpler but resets on restart and doesn't share across processes. 4. **Paste the endpoint** — I'll add the limiter exactly where it belongs, not as a generic example. --- While you answer: the fastest path if you have no Redis and just want IP-based limiting is [`slowapi`](https://github.com/laurents/slowapi) (a thin wrapper around `limits`, built for FastAPI). Two lines on the endpoint, one line of setup. Already installed in many FastAPI projects. If you **do** have Redis and need it to survive restarts or you're running multiple workers, I'll use `limits` with a Redis backend directly, or show you a one-function dependency you can `Depends()` onto any route. --- **Skipped:** middleware-level limiting, per-route custom error messages, sliding vs. fixed window choice — tell me if any of those matter.

✗ This run0.00

**What's your setup?** Before I write anything, I need to know: 1. **What identifies a "user"?** IP address, API key, JWT sub, something else? 2. **What are the limits?** (e.g. 10 req/min) 3. **Where does state live?** In-process (single server, dev) or Redis (multi-server, prod)? 4. **Show me the endpoint** — I'll wire it in directly rather than give generic code. --- While you answer: the laziest complete solution for a single-server API is `slowapi` (wraps `limits`, decorator-based, ~3 lines per endpoint). If you're already on Redis or have an existing middleware pattern, we'll use that instead.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429), no FastAPI usage

History on main

email validator | openrouter:anthropic/claude-haiku-4.5

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 2.00 → 0.00

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main2.00

```python import re def is_valid_email(email: str) -> bool: return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (full spec is huge and impractical). This catches ~99% of real addresses. If you need strict validation, use `email_validator` package. If you need to confirm the address exists, you must send a confirmation email—no regex does that. **Risk:** This accepts `a@b.c` (technically valid but rare). It rejects nothing malicious. Use this for UX feedback only; validate on the backend before trusting user data.

✗ This run0.00

Use Python's built-in `email.utils.parseaddr`: ```python from email.utils import parseaddr def is_valid_email(email): name, addr = parseaddr(email) return '@' in addr and '.' in addr.split('@')[1] ``` That's it. It parses the email, checks for `@` and a dot after it (minimal RFC 5321). For 95% of use cases, this beats a regex or a dependency. **Skipped:** RFC 5322 full compliance, length limits, DNS MX checks, plus-addressing edge cases. **Risk:** accepts some invalid addresses (e.g., `user@localhost`); if you need strict validation, ask your backend to send a confirmation link instead—that's the only real check.

Why it failed: FAIL: accepted invalid: @missing-local.com

History on main

All evals

Changed (17) All (21) Failing (4) Eval changed (15) Within chance (1) Improved (1)
FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Eval changed

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main4.50

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** — API key in header, JWT, IP address, session? 2. **What limit do you want?** — e.g. 10 requests/minute per user? 3. **Do you have Redis?** — If yes, limits survive restarts and work across multiple workers. If no, in-memory is simpler but resets on restart and doesn't share across processes. 4. **Paste the endpoint** — I'll add the limiter exactly where it belongs, not as a generic example. --- While you answer: the fastest path if you have no Redis and just want IP-based limiting is [`slowapi`](https://github.com/laurents/slowapi) (a thin wrapper around `limits`, built for FastAPI). Two lines on the endpoint, one line of setup. Already installed in many FastAPI projects. If you **do** have Redis and need it to survive restarts or you're running multiple workers, I'll use `limits` with a Redis backend directly, or show you a one-function dependency you can `Depends()` onto any route. --- **Skipped:** middleware-level limiting, per-route custom error messages, sliding vs. fixed window choice — tell me if any of those matter.

✗ This run0.00

**What's your setup?** Before I write anything, I need to know: 1. **What identifies a "user"?** IP address, API key, JWT sub, something else? 2. **What are the limits?** (e.g. 10 req/min) 3. **Where does state live?** In-process (single server, dev) or Redis (multi-server, prod)? 4. **Show me the endpoint** — I'll wire it in directly rather than give generic code. --- While you answer: the laziest complete solution for a single-server API is `slowapi` (wraps `limits`, decorator-based, ~3 lines per endpoint). If you're already on Redis or have an existing middleware pattern, we'll use that instead.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429), no FastAPI usage

History on main

email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main2.00

```python import re def is_valid_email(email: str) -> bool: return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (full spec is huge and impractical). This catches ~99% of real addresses. If you need strict validation, use `email_validator` package. If you need to confirm the address exists, you must send a confirmation email—no regex does that. **Risk:** This accepts `a@b.c` (technically valid but rare). It rejects nothing malicious. Use this for UX feedback only; validate on the backend before trusting user data.

✗ This run0.00

Use Python's built-in `email.utils.parseaddr`: ```python from email.utils import parseaddr def is_valid_email(email): name, addr = parseaddr(email) return '@' in addr and '.' in addr.split('@')[1] ``` That's it. It parses the email, checks for `@` and a dot after it (minimal RFC 5321). For 95% of use cases, this beats a regex or a dependency. **Skipped:** RFC 5322 full compliance, length limits, DNS MX checks, plus-addressing edge cases. **Risk:** accepts some invalid addresses (e.g., `user@localhost`); if you need strict validation, ask your backend to send a confirmation link instead—that's the only real check.

Why it failed: FAIL: accepted invalid: @missing-local.com

History on main

CSV sum | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.50

```python import pandas as pd df = pd.read_csv('sales.csv') total = df['amount'].sum() print(total) ``` That's it. If pandas is already a project dependency, use it—it's three lines and handles parsing, missing values, and type conversion. If you need the total as a return value instead of printed, wrap it in a function. **Skipped:** error handling for missing file, type validation, filtering rows. **Risk:** if 'amount' has non-numeric values, `.sum()` will skip them silently (pandas default)—add `.astype(float)` before `.sum()` if that's a problem.

✓ This run1.00

```python import pandas as pd df = pd.read_csv('sales.csv') total = df['amount'].sum() print(total) ``` **Skipped:** error handling (file not found, missing column), type checking, logging. **Risk:** if the file has no 'amount' column or is malformed, it crashes silently; add a try/except if this runs unattended.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.50

```python import csv with open("sales.csv", newline="") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(total) ``` Skipped: error handling for missing file/bad values (not asked). Risk: assumes a header row named `amount` and that every value parses as a number—one bad cell raises.

✓ This run1.00

```python import csv with open('sales.csv', newline='') as f: total = sum(float(row['amount']) for row in csv.DictReader(f)) print(total) ``` Skipped: error handling for missing file/column/bad values, and didn't verify your column is named exactly `amount`. Will crash if any `amount` cell is empty or non-numeric.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on empty `amount` cells — replace `float(...)` with `float(row["amount"] or 0)` if needed.

✓ This run1.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on non-numeric `amount` cells.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main9.50

# Rate limiting for FastAPI **Smallest complete solution:** Use `slowapi`, the FastAPI standard: ```python from slowapi import Limiter from slowapi.util import get_remote_address limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): return {"data": "value"} ``` **Install:** ```bash pip install slowapi ``` **Add to exception handlers** (one time, in your main app file): ```python from slowapi.errors import RateLimitExceeded from fastapi.responses import JSONResponse @app.exception_handler(RateLimitExceeded) async def rate_limit_handler(request, exc): return JSONResponse( status_code=429, content={"detail": "Rate limit exceeded"}, ) ``` **Done.** Apply `@limiter.limit("N/timeunit")` to any endpoint. Timeunits: `second`, `minute`, `hour`, `day`. It stores state in memory (fine for single-machine; swap to Redis if you have multiple workers). --- **Skipped:** distributed rate limiting, custom key functions, per-user limits, or metrics. **Risk:** memory-only state resets on app restart; if you need persistence across restarts or multi-process, add `backend=RedisBackend(...)` to the Limiter.

✓ This run1.00

# Rate limiting for FastAPI **Smallest complete solution:** ```python from fastapi_limiter2 import FastAPILimiter from fastapi_limiter2.depends import RateLimiter from fastapi import FastAPI app = FastAPI() @app.on_event("startup") async def startup(): await FastAPILimiter.init(redis_connection) # or in-memory store @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): return {"data": "..."} ``` **If you don't have Redis yet:** use the simpler in-memory option: ```python from slowapi import Limiter from slowapi.util import get_remote_address limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): return {"data": "..."} ``` Install: `pip install slowapi` (no external service needed). **What I skipped:** custom key functions (IP vs user ID), distributed rate limiting across servers, response headers with retry info. Name the rate (requests/time window) that fits your use case and I'll adjust.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main5.50

Use `slowapi` — it's the standard FastAPI rate limiter, a few lines. Install: `pip install slowapi` ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") async def your_endpoint(request: Request): # `request` param is required by slowapi ... ``` Change `5/minute` to whatever "spam" means for you. Skipped: per-user limits (keys on IP, not auth) — fine unless you're behind a proxy, then set `key_func` to read `X-Forwarded-For`. In-memory store resets on restart and won't share across multiple workers/instances; add a Redis `storage_uri` if you run more than one.

✓ This run1.00

Use `slowapi` — it's the standard, battle-tested FastAPI rate limiter. A few lines, no custom bucket code. ```python # pip install slowapi from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") # tune to taste async def your_endpoint(request: Request): # `request` param is required ... ``` Two gotchas worth knowing: - The endpoint **must** take `request: Request` or slowapi can't read the client IP. - `get_remote_address` uses the socket IP. Behind a proxy/load balancer that's the proxy's IP (everyone shares one limit). If you're behind one, set the key to read `X-Forwarded-For` — but only if you trust the proxy to set it, since clients can forge it. Skipped: per-user limits (needs your auth) and a shared Redis backend (slowapi defaults to in-memory, so limits reset on restart and aren't shared across multiple workers/instances). If you run more than one worker, say so and I'll add `storage_uri="redis://..."`.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // Only this fires, 300ms later ``` --- **Skipped:** cancellation method, trailing/leading edge options, return value from fn, max-wait limit. Add any if you need them. **Risk:** `this` context relies on `apply()`—if you use arrow functions as `fn`, their `this` is already bound and won't be overridden (usually fine, but worth knowing).

✓ This run1.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` Done. Handles `this` binding, passes all arguments, clears the old timer on each call so it only fires after `delay` ms of silence. **Test it:** ```javascript const log = debounce(console.log, 300); log('a'); log('b'); log('c'); // after 300ms: logs 'c' once ``` --- **Skipped:** cancellation method (return `.cancel()` to clear the timer), leading/trailing edge options, return value from `fn`. Add only if you need them. **Risk:** if `fn` throws, the error fires async in the timeout—add `.catch()` or `try`/`catch` in `fn` if that matters.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js let calls = 0; const d = debounce(() => calls++, 50); d(); d(); d(); setTimeout(() => console.assert(calls === 1, `expected 1, got ${calls}`), 100); ``` Skipped: leading-edge, cancel/flush methods, and a return value (debounced calls are async, so there's nothing to return). Say the word if you need any of those.

✓ This run1.00

```js function debounce(fn, delay) { let t; return function (...args) { clearTimeout(t); t = setTimeout(() => fn.apply(this, args), delay); }; } ``` `this`/`args` are preserved so it works on methods and event handlers. Skipped: `cancel`/`flush` methods and leading-edge option (not asked). No return value from the debounced call — add a promise wrapper only if a caller needs it.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Self-check** (runs once in Node or a browser console): ```js // quick smoke-test — remove in production let calls = 0; const inc = debounce(() => calls++, 50); inc(); inc(); inc(); setTimeout(() => console.assert(calls === 1, "debounce failed"), 100); ``` **Skipped:** leading-edge ("immediate") option, cancel/flush methods — not asked for. **Risk:** `this` binding relies on normal `function` syntax; arrow-function callers won't need it, but class method callers will — works correctly either way.

✓ This run1.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Self-check** ```js // node -e to verify basic behaviour let calls = 0; const inc = debounce(() => calls++, 50); inc(); inc(); inc(); setTimeout(() => console.assert(calls === 1, 'expected 1 call, got ' + calls), 100); ``` --- Skipped: leading-edge (immediate) option, cancel/flush methods — add if needed. Risk: `this` binding works for regular calls and methods; won't help if the caller already lost `this` before passing the function in.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main7.00

# Countdown Timer ```jsx export function CountdownTimer({ seconds, onComplete }) { const [timeLeft, setTimeLeft] = useState(seconds); useEffect(() => { if (timeLeft <= 0) { onComplete?.(); return; } const timer = setInterval(() => setTimeLeft(t => t - 1), 1000); return () => clearInterval(timer); }, [timeLeft, onComplete]); return <div>{timeLeft}s</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={60} onComplete={() => console.log('done')} /> ``` --- **Skipped:** reset button, pause, display format (mm:ss), animations, styling, aria labels, effect cleanup on unmount edge case. **Risk:** `onComplete` in the dependency array could fire multiple times if you pass a new function every render—wrap it in `useCallback` if that's your pattern. If `seconds` changes mid-countdown, it restarts.

✓ This run1.00

# Countdown Timer ```jsx export function CountdownTimer({ seconds, onComplete }) { const [remaining, setRemaining] = React.useState(seconds); React.useEffect(() => { if (remaining <= 0) { onComplete?.(); return; } const timer = setInterval(() => setRemaining(r => r - 1), 1000); return () => clearInterval(timer); }, [remaining, onComplete]); return <div>{remaining}</div>; } ``` **What I skipped:** pause/resume buttons, custom formatting, sound, animations. **Risk:** `onComplete` in the dependency array could fire repeatedly if it's not memoized; pass a stable function or remove the callback if you don't need it.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main7.00

```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { setRemaining(seconds); if (seconds <= 0) return; const id = setInterval(() => { setRemaining((r) => (r <= 1 ? (clearInterval(id), 0) : r - 1)); }, 1000); return () => clearInterval(id); }, [seconds]); return <span>{remaining}</span>; } ``` Skipped (nobody asked): pause/reset, mm:ss formatting, onComplete callback — add if you need them. Risk: `setInterval` drifts over long runs; fine for a display timer, not for anything billed by the second.

✓ This run1.00

```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { setRemaining(seconds); if (seconds <= 0) return; const id = setInterval(() => { setRemaining((r) => (r <= 1 ? (clearInterval(id), 0) : r - 1)); }, 1000); return () => clearInterval(id); }, [seconds]); return <span>{remaining}</span>; } ``` Skipped (nobody asked): start/pause/reset controls, mm:ss formatting, onComplete callback — add later if you need them. Risk: `setInterval` drifts over long runs; fine for short timers, use timestamp math if you need accuracy past a few minutes.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main13.50

Here's the minimal component: ```jsx // CountdownTimer.jsx import { useState, useEffect, useRef } from "react"; export default function CountdownTimer({ seconds, onComplete }) { const [remaining, setRemaining] = useState(seconds); const intervalRef = useRef(null); useEffect(() => { if (remaining <= 0) { onComplete?.(); return; } intervalRef.current = setInterval(() => { setRemaining((r) => { if (r <= 1) { clearInterval(intervalRef.current); onComplete?.(); return 0; } return r - 1; }); }, 1000); return () => clearInterval(intervalRef.current); }, []); // run once on mount const m = Math.floor(remaining / 60).toString().padStart(2, "0"); const s = (remaining % 60).toString().padStart(2, "0"); return <div aria-live="polite">{m}:{s}</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={90} onComplete={() => alert("Done!")} /> ``` --- **Skipped:** pause/resume/reset controls, resetting when `seconds` prop changes after mount, and any visual styling — add those if you need them. **Risk:** the interval drifts slightly over long durations (~1 s per few minutes); use `Date`-based logic if precision matters.

✓ This run1.00

Here's the minimum that works: ```jsx import { useState, useEffect } from "react"; export default function CountdownTimer({ seconds }) { const [timeLeft, setTimeLeft] = useState(seconds); useEffect(() => { if (timeLeft <= 0) return; const id = setInterval(() => setTimeLeft((t) => t - 1), 1000); return () => clearInterval(id); }, [timeLeft]); return <div>{timeLeft}s</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={60} /> ``` That's it. `useEffect` re-registers a fresh interval each tick — slightly less efficient than `setTimeout` but dead simple and correct. --- **Skipped:** pause/resume, reset, onComplete callback, formatting as MM:SS, styling. Add `onComplete` prop and call it inside the `if (timeLeft <= 0)` block if you need it. **Risk:** `seconds` prop changes after mount are ignored — `timeLeft` initialises once. Wrap in a `key` prop at the call site to reset: `<CountdownTimer key={seconds} seconds={seconds} />`.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-opus-4.8 Too early to tell

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main4.50

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.fullmatch(email)) ``` Quick self-check: ```python assert is_valid_email("a@b.co") assert not is_valid_email("a@b") # no TLD assert not is_valid_email("a b@c.co") # space assert not is_valid_email("a@@b.co") # double @ ``` Skipped: full RFC 5322 (quoted locals, IP literals, comments) — nobody needs it and the regex gets monstrous. Risk: this is a format check, not a deliverability check; the only way to know an address works is to send to it.

✓ This run1.00

```python import re _EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.match(email)) ``` ```python # self-check assert is_valid_email("a@b.co") assert not is_valid_email("a@b") assert not is_valid_email("a b@c.com") assert not is_valid_email("@b.com") ``` Skipped: full RFC 5322 compliance (hundreds of lines, rejects some valid addresses anyway). Risk: this accepts syntactically odd but rare-valid forms and can't tell if the mailbox actually exists — the only real check is sending a confirmation email. If you need that, say so.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-sonnet-4.6 Too early to tell

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main3.50

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.fullmatch(email)) ``` **Usage** ```python is_valid_email("user@example.com") # True is_valid_email("bad@") # False ``` --- Skipped: RFC 5322 edge cases (quoted locals, IP literals, comments) — real-world emails don't use them. Risk: rejects technically valid but exotic addresses like `"user name"@example.com`; use a library like [`email-validator`](https://pypi.org/project/email-validator/) if you need strict RFC compliance or MX-record checking.

✓ This run1.00

```python import re # RFC 5322 simplified – covers 99 % of real addresses; rejects exotic edge cases EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$") def is_valid_email(email: str) -> bool: return bool(EMAIL_RE.match(email)) ``` **Usage** ```python is_valid_email("user@example.com") # True is_valid_email("bad@") # False ``` --- **Skipped:** quoted local parts (`"foo bar"@example.com`), IP-address domains, and Unicode TLDs – all valid per RFC but vanishingly rare in practice. **Risk:** the only real validation is sending a confirmation email; regex can't prove the address exists or accepts mail.

Why it failed: All assertions passed

History on main

hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail Within chance

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

✓ main0.67

Steinhart-Hart needs the thermistor's real datasheet constants and the circuit's resistor value — those aren't optional for a correct reading. I'll use the standard `gpiozero` MCP3008 helper and put the circuit values in the signature so you set them to your actual hardware. ```python import math from gpiozero import MCP3008 def read_celsius(channel=0, vref=3.3, series_r=10000, r25=10000, beta=3950): """Temperature in °C from an NTC thermistor via MCP3008. Assumes thermistor between Vref and the ADC pin, series_r to ground. Set r25/beta from your thermistor's datasheet, series_r to your resistor. """ v = MCP3008(channel=channel).value # 0.0–1.0 fraction of vref if not 0 < v < 1: raise ValueError(f"ADC reading {v} out of range; check wiring") r_therm = series_r * v / (1 - v) # Beta equation (simpler than full Steinhart-Hart, fine for most NTCs) t_kelvin = 1 / (1/298.15 + math.log(r_therm / r25) / beta) return t_kelvin - 273.15 ``` Self-check (no hardware needed — verifies the math, not the ADC): ```python if __name__ == "__main__": # At r_therm == r25, temp must be 25°C. v for that: series_r*v/(1-v)=r25 # with series_r==r25 -> v=0.5 class _Fake: value = 0.5 import gpiozero; gpiozero.MCP3008 = lambda channel: _Fake() assert abs(read_celsius() - 25.0) < 0.01 print("ok") ``` Skipped: full Steinhart-Hart (Beta eq. is ±0.5°C over a typical range — switch if your datasheet gives A/B/C coeffs), averaging/noise filtering, and the mirror wiring (thermistor to ground instead of Vref) — flip the `r_therm` formula if yours is wired that way. Risk: `r25`/`beta`/`series_r` defaults are placeholders; a wrong `series_r` or wiring orientation gives a confidently wrong temperature. Confirm against your hardware before trusting it.

✗ This run0.00

```python from gpiozero import MCP3008 from math import log # Vishay NTCLE100E3103 10k thermistor, Beta model. Adjust B/R0/T0 for your part. B = 3977.0 # Beta (K), from datasheet R0 = 10000.0 # resistance at T0 (ohms) T0 = 298.15 # 25 C in Kelvin R_SERIES = 10000.0 # fixed divider resistor (ohms) def read_temperature_c(channel=0): """Celsius from an NTC thermistor on an MCP3008. Thermistor to 3.3V, R_SERIES to GND, junction to ADC.""" v = MCP3008(channel=channel).value # 0.0-1.0 fraction of Vref if not 0.0 < v < 1.0: raise ValueError(f"ADC reading {v} at rail; check wiring") r_therm = R_SERIES * v / (1.0 - v) # divider: thermistor on the high side inv_t = 1.0 / T0 + log(r_therm / R0) / B return 1.0 / inv_t - 273.15 ``` Self-check (runs without hardware): ```python if __name__ == "__main__": # At R_therm == R0, voltage fraction == 0.5, temp must be T0 (25 C) import gpiozero, math r = R0 inv_t = 1.0 / T0 + math.log(r / R0) / B assert abs((1.0 / inv_t - 273.15) - 25.0) < 0.01 print("ok") ``` Skipped: averaging/smoothing, Steinhart–Hart (Beta is good to ~±1°C over a modest range; upgrade to S-H coefficients if you need wide-range accuracy). Risk: the constants and divider topology are for a specific part/wiring — wrong `B`/`R0`/`R_SERIES` or a thermistor on the low side gives silently wrong readings. Set them from your datasheet.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) Improved

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

✗ main0.00

Using the standard `gpiozero` MCP3008 helper — it's the common Pi ADC library and avoids hand-rolling SPI. ```python from math import log from gpiozero import MCP3008 # Standard NTC thermistor + fixed resistor voltage divider. # Defaults are a common 10k NTC with Beta=3950, 10k series resistor. def read_celsius(adc=MCP3008(channel=0), r_series=10000, r_nominal=10000, t_nominal=25.0, beta=3950): v = adc.value # 0.0..1.0 fraction of Vref if not 0.0 < v < 1.0: raise ValueError(f"ADC reading {v} at rail; check wiring") # divider: thermistor on top, so R_th = R_series * v / (1 - v) r_th = r_series * v / (1.0 - v) # Beta equation, returns Kelvin -> Celsius t_kelvin = 1.0 / (1.0/(t_nominal + 273.15) + log(r_th/r_nominal)/beta) return t_kelvin - 273.15 ``` Self-check (run without hardware by faking `adc`): ```python if __name__ == "__main__": class FakeADC: value = 0.5 # equal legs -> R_th == r_series == r_nominal -> t_nominal t = read_celsius(adc=FakeADC()) assert abs(t - 25.0) == 0 or abs(t - 25.0) < 1e-6, t print("ok", t) ``` At v=0.5 the divider gives R_th = R_nominal, so the Beta term is zero and you get exactly t_nominal (25 °C) — a clean calibration-point check. Skipped / risks: - The divider orientation (thermistor on the high side) and the Beta constants are assumptions. If your thermistor is on the ground side, swap to `r_series * (1-v)/v`. Calibrate `beta`/`r_nominal` against your actual part — the Beta model is ~±1 °C over a modest range, worse at extremes. - `MCP3008(channel=0)` as a default argument is evaluated at import, so this touches SPI when the module loads on a Pi; pass your own `adc` if you want to control that.

✗ This run0.33

Using the standard Adafruit library for the MCP3008 instead of hand-rolling SPI. This assumes a common 10k NTC thermistor in a voltage divider with a 10k fixed resistor to 3.3V — those values are wired into the math via the Beta equation. ```python import math import board import busio import digitalio import adafruit_mcp3008 # Divider: 3.3V -- R_FIXED -- node(ADC ch0) -- NTC -- GND VREF = 3.3 R_FIXED = 10000.0 # fixed resistor, ohms R0 = 10000.0 # thermistor resistance at 25C T0 = 298.15 # 25C in Kelvin BETA = 3950.0 # thermistor beta coefficient (check your datasheet) spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) _mcp = adafruit_mcp3008.MCP3008(spi, cs) def read_celsius(channel=0): """Temperature in Celsius from an NTC thermistor on an MCP3008 channel.""" raw = _mcp.read_adc(channel) # 0..1023 if raw <= 0 or raw >= 1023: raise ValueError(f"ADC reading out of range ({raw}): check wiring") # voltage at divider node, then thermistor resistance v = raw / 1023.0 * VREF r_therm = R_FIXED * v / (VREF - v) # Beta equation inv_t = 1.0 / T0 + math.log(r_therm / R0) / BETA return 1.0 / inv_t - 273.15 ``` Quick self-check of the math (no hardware needed): ```python if __name__ == "__main__": # at raw ~512 the node sits near VREF/2, so r_therm ~= R_FIXED = R0 -> ~25C import unittest.mock as m _mcp.read_adc = lambda ch: 512 t = read_celsius() assert 24 < t < 26, t print(f"OK: {t:.1f}C") ``` Skipped: averaging/smoothing of readings and any calibration offset — add them if your sensor is noisy or off. Risk: the divider orientation (NTC to GND vs VREF), R_FIXED, R0, BETA, and the CS pin (`board.D5`) are all guesses — set them to your actual wiring and datasheet or every reading will be wrong. Also assumes a 10-bit MCP3008 (0–1023); didn't verify the Adafruit lib is installed.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

one runnable check | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / one runnable check | ponytail Unchanged

Input

{"probe":"onecheck","task":"Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."}

✓ main1.00

```python import re def parse_duration(s): total = 0 for value, unit in re.findall(r'(\d+)([hms])', s): total += int(value) * {'h': 3600, 'm': 60, 's': 1}[unit] return total assert parse_duration('1h30m45s') == 5445 assert parse_duration('45s') == 45 assert parse_duration('') == 0 ``` Skipped: input validation (garbage like `'1x'` is silently ignored, not rejected) and fractional/day/ms units — none were asked for. Risk: if you need to reject malformed input, this won't; add an upfront `re.fullmatch` check.

✓ This run1.00

```python import re def parse_duration(s): units = {'h': 3600, 'm': 60, 's': 1} matches = re.findall(r'(\d+)([hms])', s) if not matches: raise ValueError(f"invalid duration: {s!r}") return sum(int(n) * units[u] for n, u in matches) assert parse_duration('1h30m45s') == 5445 assert parse_duration('90m') == 5400 assert parse_duration('45s') == 45 ``` Skipped: fractional values ('1.5h'), days/ms, and strict ordering/duplicate checks — `'30m1h'` and `'1h1h'` parse happily. Add those if your input isn't trusted.

History on main

one runnable check | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / one runnable check | ponytail (AGENTS.md) Unchanged

Input

{"probe":"onecheck","task":"Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."}

✓ main1.00

```python import re def parse_duration(s): units = {'h': 3600, 'm': 60, 's': 1} if not re.fullmatch(r'(\d+[hms])+', s): raise ValueError(f"invalid duration: {s!r}") return sum(int(n) * units[u] for n, u in re.findall(r'(\d+)([hms])', s)) assert parse_duration('1h30m45s') == 5445 assert parse_duration('45s') == 45 assert parse_duration('2h') == 7200 ``` Skipped: fractional values (`1.5h`), days/ms, and duplicate-unit rejection (`1h1h` parses as 2h). Risk: input with no unit or stray chars raises `ValueError` — catch it if your caller feeds untrusted strings.

✓ This run1.00

```python import re def parse_duration(s): total = 0 for value, unit in re.findall(r'(\d+)([hms])', s): total += int(value) * {'h': 3600, 'm': 60, 's': 1}[unit] return total assert parse_duration('1h30m45s') == 5445 assert parse_duration('45s') == 45 assert parse_duration('') == 0 ``` Skipped: days/weeks, fractional values, input validation (garbage between tokens is silently ignored). Risk: `'5x'` returns 0, not an error — add a strict check if callers can pass untrusted strings.

History on main

requested explanation | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / requested explanation | ponytail Unchanged

Input

{"probe":"explanation","task":"Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] \u003e 0:\n r.append(x['a'] * 2)\n return r"}

✓ main1.00

Lazy read: the function keeps items where `a` exists and is positive, doubling each. ```python def p(d): return [x['a'] * 2 for x in d if x.get('a', 0) > 0] ``` **Changes:** 1. Loop + `append` → list comprehension. Same result, one line, standard Python idiom. 2. `x.get('a') and x['a'] > 0` → `x.get('a', 0) > 0`. The original checked truthiness then compared; since the keep condition is `> 0`, `get('a', 0)` folds both into one check (missing key defaults to `0`, which fails `> 0`). **Edge case note:** original `x.get('a') and ...` treats `a == 0` and `a is None` the same (both skipped); so does mine. But if `a` could be a non-number (e.g. a string), both versions behave identically on the comparison — no change there. Skipped: renaming `p`/`d`/`x` to meaningful names and adding type hints — you asked for readability but I don't know the domain, so I left names alone rather than guess wrong. Didn't test; behavior is identical by inspection.

✓ This run1.00

```python def p(d): return [x['a'] * 2 for x in d if x.get('a', 0) > 0] ``` That's the whole refactor. Renaming `p`, `d`, `x` would help more, but I don't know what they mean, so I left them. ## What I changed and why 1. **Loop → list comprehension.** The original builds a list by appending in a loop; a comprehension says "list of these, for these, where this" in one line. Same result, less scaffolding. 2. **`x.get('a') and x['a'] > 0` → `x.get('a', 0) > 0`.** The original checks the key exists *and* is truthy *and* is `> 0` — but `> 0` already implies truthy and non-None-safe if we default the missing case. `get('a', 0)` returns `0` when the key is absent, and `0 > 0` is `False`, so missing keys are filtered out just the same. **Behavior note / one risk:** these two aren't identical on edge cases. If `x['a']` can be a non-number (string, None stored explicitly), the original's `and` short-circuits differently than a bare `> 0` comparison — and in Python 3 comparing e.g. `"5" > 0` raises `TypeError`. The original would also raise on a string (`"5" > 0`), so I didn't make it worse, but if `'a'` can hold non-numbers you want an explicit type check instead. Skipped: renaming (meanings unknown) and a test (don't know the intended contract). If you tell me what `p` does, I'll name it and drop in a one-line assert.

History on main

requested explanation | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / requested explanation | ponytail (AGENTS.md) Unchanged

Input

{"probe":"explanation","task":"Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] \u003e 0:\n r.append(x['a'] * 2)\n return r"}

✓ main1.00

No code to read against, no callers shown, so this is a pure cosmetic pass. Here it is: ```python def double_positive_a(items): return [item['a'] * 2 for item in items if item.get('a', 0) > 0] ``` Step-by-step: 1. **`p` → `double_positive_a`**: the name now says what it does (double the `'a'` field where positive). 2. **`d` → `items`**: it's a list of dicts, so a plural reads better. `x` → `item`. 3. **Collapsed the loop into a comprehension**: building a list by appending in a loop is exactly what a list comprehension is for. Fewer lines, same result. 4. **`x.get('a') and x['a'] > 0` → `item.get('a', 0) > 0`**: the original guards against `None`/missing (`.get('a')` is falsy) *and* checks `> 0`. A missing key defaulting to `0` fails `> 0` the same way, so one condition covers both. One lookup instead of two. One behavior note: the original treated `'a': 0` as falsy and skipped it; `> 0` skips it too, so that matches. If `'a'` could be a non-number (string, `None` stored explicitly as a value), both versions would behave the same up to the `> 0` comparison — `None > 0` raises in Python 3, and it did in the original too, so no change there. Skipped: no test added (trivial one-liner, behavior preserved). Risk: if any `'a'` value is a non-comparable type you'll get a `TypeError` — same as before, so not introduced by this change.

✓ This run1.00

Here's the refactor: ```python def doubled_positive_amounts(items): return [item['a'] * 2 for item in items if item.get('a', 0) > 0] ``` Step-by-step: 1. **`p` → `doubled_positive_amounts`**: the name now says what it returns. I guessed `'a'` means "amount"; rename the key too if you know what it is. 2. **`d` → `items`, `x` → `item`**: readable loop variable, plural for the collection. 3. **Merged the manual loop + append into a list comprehension**: same result, one line, the standard Python idiom for "filter then map." 4. **`x.get('a') and x['a'] > 0` → `item.get('a', 0) > 0`**: the original checks truthiness (`get('a')`) *then* re-reads `x['a']`. The `get('a', 0)` default handles the missing/`None`/falsy case and does the positive check in one expression — same behavior, no double lookup. One behavioral note: the original's `x.get('a') and ...` would skip a non-numeric truthy value before comparing, while mine passes it straight to `> 0`. Given the values are multiplied by 2, they're meant to be numbers, so this difference only matters for malformed data. Skipped: didn't rename the `'a'` dict key (don't know the real domain meaning) and didn't add type hints or a test — trivial change, and I don't know the real key/value types. Risk: if your data can contain non-numeric truthy values under `'a'`, mine raises `TypeError` where the original silently skipped; add an `isinstance(item.get('a'), (int, float))` guard if that's possible.

History on main