7beb065 on main | evalship

Notes

Couldn't run: 1 eval in Ponytail vs caveman vs no-skill failed before it was scored (ModuleNotFoundError: No module named 'email_validator'), so it isn't compared with main@545983b.

1 eval broke in this commit

email validator | openrouter:anthropic/claude-haiku-4.5

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 1.00 → 0.00

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main1.00

```python import re def is_valid_email(email): return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (overkill for 99% of uses), async DNS lookups, international characters, length limits, or a dependency like `email-validator`. This regex catches typos and obvious mistakes. **Risk:** The regex is permissive—it accepts `a@b.c` and `user@domain.co.uk` but not `user@localhost`. If you need to reject fakes or validate international emails, say so and I'll add it.

✗ This run0.00

**Don't write one.** ```python from email_validator import validate_email def is_valid_email(email: str) -> bool: try: validate_email(email, check_deliverability=False) return True except: return False ``` Install: `pip install email-validator` This handles the RFC 5321/5322 edge cases (dots, plus signs, quoted strings, internationalized domains) that a regex breaks on. The standard library's `email.utils` is too lenient. **Skipped:** regex, length limits, domain checks, async verification. **Risk:** `check_deliverability=False` means no SMTP lookup—fine for signup forms, not for critical mail systems.

Why it failed: Traceback (most recent call last): File "/tmp/ponytail-bench-1793293473270-yz29yr5mt3.py", line 2, in <module> from email_validator import validate_email ModuleNotFoundError: No module named 'email_validator'

History on main

All evals

EvalChangemainThis run

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Now passing ✗ 0.00 ✓ 1.00

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main0.00

**What's your setup?** The right answer depends on two things: 1. **Single server or multiple?** - Single → in-process counter is enough - Multiple → need shared state (Redis, etc.) 2. **Do you already have Redis / a cache layer installed?** --- I'll assume **single server, no Redis** — the most common case for a service that hasn't needed rate limiting yet. **Install one thing:** ```bash pip install slowapi ``` `slowapi` wraps `limits` and plugs into FastAPI/Starlette with almost no code. --- **Minimal wiring** (add to wherever your app is created): ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) ``` **On the endpoint:** ```python @app.get("/your-endpoint") @limiter.limit("20/minute") # ← change to taste async def your_endpoint(request: Request): ... ``` `Request` must be a parameter even if you don't use it — slowapi needs it. --- **That's it.** The handler returns a proper `429 Too Many Requests` automatically. **Limit string examples:** `"5/second"`, `"100/hour"`, `"1000/day"` --- **Skipped:** auth-based limiting (per user/API key instead of per IP) — IP limiting is wrong if many users share a NAT. If that's your situation, swap `get_remote_address` for a function that pulls the user ID from your auth token. **Risk:** in-memory state resets on restart and doesn't share across workers. If you run multiple `uvicorn` workers (`--workers 4`), each gets its own counter — effective limit becomes `N × your limit`. Fix by adding `storage_uri="redis://localhost"` to `Limiter(...)` when you're ready.

✓ This run1.00

**What's your setup?** The right answer depends on two things: 1. **Single server or multiple?** 2. **Do you already have Redis installed?** --- ## Single server (no Redis needed) Use `slowapi` — it wraps `limits` and plugs into FastAPI with almost no code. ```bash pip install slowapi ``` ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded from fastapi import FastAPI, Request limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("20/minute") async def your_endpoint(request: Request): # request param required by slowapi ... ``` Returns `429 Too Many Requests` automatically. --- ## Multiple servers (needs shared state) Use Redis so all instances share the counter: ```python limiter = Limiter(key_func=get_remote_address, storage_uri="redis://localhost:6379") ``` Everything else stays the same. --- ## Picking your limit | Endpoint type | Starting point | |---|---| | Auth / login | `5/minute` | | Search / read | `60/minute` | | Write / submit | `20/minute` | | File upload | `10/minute` | --- **Skipped:** per-user limits (requires auth identity as the key — swap `get_remote_address` for a function that returns `request.state.user_id`), rate limit headers in the response, and warming/burst allowances. **Risk:** IP-based limiting breaks behind a shared NAT or proxy where many users share one IP — consider user-ID keying if that's your audience.

Why it failed: All assertions passed

History on main

evalship-hq / ponytail

Commit 7beb065 on main

Notes

1 eval broke in this commit

email validator | openrouter:anthropic/claude-haiku-4.5

All evals