Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Commit 545983b on main

6 days ago · vs main@7e38f14 · evals run

evals: 20 / 21
Ponytail vs ca...14 / 15+1Ponytail behav...6 / 6+2

All evals

Changed (3) All (21) Failing (1) Now passing (3)
FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Unchanged

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main0.00

**What's your setup?** Before I write anything, I need to know: 1. **What identifies a "user"?** IP address, API key, JWT sub, something else? 2. **What are the limits?** (e.g. 10 req/min) 3. **Where does state live?** In-process (single server, dev) or Redis (multi-server, prod)? 4. **Show me the endpoint** — I'll wire it in directly rather than give generic code. --- While you answer: the laziest complete solution for a single-server API is `slowapi` (wraps `limits`, decorator-based, ~3 lines per endpoint). If you're already on Redis or have an existing middleware pattern, we'll use that instead.

✗ This run0.00

**What's your setup?** The right answer depends on two things: 1. **Single server or multiple?** - Single → in-process counter is enough - Multiple → need shared state (Redis, etc.) 2. **Do you already have Redis / a cache layer installed?** --- I'll assume **single server, no Redis** — the most common case for a service that hasn't needed rate limiting yet. **Install one thing:** ```bash pip install slowapi ``` `slowapi` wraps `limits` and plugs into FastAPI/Starlette with almost no code. --- **Minimal wiring** (add to wherever your app is created): ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) ``` **On the endpoint:** ```python @app.get("/your-endpoint") @limiter.limit("20/minute") # ← change to taste async def your_endpoint(request: Request): ... ``` `Request` must be a parameter even if you don't use it — slowapi needs it. --- **That's it.** The handler returns a proper `429 Too Many Requests` automatically. **Limit string examples:** `"5/second"`, `"100/hour"`, `"1000/day"` --- **Skipped:** auth-based limiting (per user/API key instead of per IP) — IP limiting is wrong if many users share a NAT. If that's your situation, swap `get_remote_address` for a function that pulls the user ID from your auth token. **Risk:** in-memory state resets on restart and doesn't share across workers. If you run multiple `uvicorn` workers (`--workers 4`), each gets its own counter — effective limit becomes `N × your limit`. Fix by adding `storage_uri="redis://localhost"` to `Limiter(...)` when you're ready.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429), no FastAPI usage

History on main