Rate-limit check: read every Python block, and no time words in strings #6
gglucass · fix-ratelimit-check → main · 8ec83e5 ·
6 days ago · vs main@545983b
· evals run
Notes
-
Not comparable: The ratelimit scorer in correctness.js now reads every Python block and strips strings before the time check. The description says this flips Sonnet 4.6's FastAPI rate-limit answer from fail to pass, and that case is in promptfooconfig.yaml (line 47), so base scores for it won't compare. Treat this run as the new baseline for that case (
benchmarks/correctness.js:255).
All evals
FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Now passing
Input
{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}
✗ main0.00
**What's your setup?** The right answer depends on two things: 1. **Single server or multiple?** - Single → in-process counter is enough - Multiple → need shared state (Redis, etc.) 2. **Do you already have Redis / a cache layer installed?** --- I'll assume **single server, no Redis** — the most common case for a service that hasn't needed rate limiting yet. **Install one thing:** ```bash pip install slowapi ``` `slowapi` wraps `limits` and plugs into FastAPI/Starlette with almost no code. --- **Minimal wiring** (add to wherever your app is created): ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) ``` **On the endpoint:** ```python @app.get("/your-endpoint") @limiter.limit("20/minute") # ← change to taste async def your_endpoint(request: Request): ... ``` `Request` must be a parameter even if you don't use it — slowapi needs it. --- **That's it.** The handler returns a proper `429 Too Many Requests` automatically. **Limit string examples:** `"5/second"`, `"100/hour"`, `"1000/day"` --- **Skipped:** auth-based limiting (per user/API key instead of per IP) — IP limiting is wrong if many users share a NAT. If that's your situation, swap `get_remote_address` for a function that pulls the user ID from your auth token. **Risk:** in-memory state resets on restart and doesn't share across workers. If you run multiple `uvicorn` workers (`--workers 4`), each gets its own counter — effective limit becomes `N × your limit`. Fix by adding `storage_uri="redis://localhost"` to `Limiter(...)` when you're ready.
✓ This PR1.00
**What's your setup?** The right answer depends on: 1. **Single instance or multiple?** (single server vs. load-balanced / multiple workers) 2. **Do you have Redis?** (needed for multi-instance) 3. **What's the limit?** (e.g. 10 req/min per IP?) --- In the meantime, here's the **smallest complete solution for a single instance** using nothing but what FastAPI already gives you — no new dependencies: ```python from fastapi import FastAPI, Request, HTTPException from collections import defaultdict import time app = FastAPI() # shortcut: in-memory, resets on restart, not shared across workers; use Redis-backed slowapi when you scale out _rate_limit: dict[str, list[float]] = defaultdict(list) LIMIT = 10 # requests WINDOW = 60.0 # seconds def check_rate_limit(request: Request): key = request.client.host now = time.monotonic() timestamps = _rate_limit[key] # drop timestamps outside the window _rate_limit[key] = [t for t in timestamps if now - t < WINDOW] if len(_rate_limit[key]) >= LIMIT: raise HTTPException(status_code=429, detail="Too many requests") _rate_limit[key].append(now) @app.get("/your-endpoint") def your_endpoint(request: Request): check_rate_limit(request) return {"message": "ok"} ``` --- If you **do need multi-instance**, the only dependency worth adding is [`slowapi`](https://github.com/laurentS/slowapi) (thin wrapper around `limits`, battle-tested): ```bash pip install slowapi ``` ```python from fastapi import FastAPI, Request from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) # swap in Redis storage_uri= when ready app = FastAPI() app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): return {"message": "ok"} ``` To back it with Redis, add one arg: `Limiter(key_func=get_remote_address, storage_uri="redis://localhost:6379")`. --- **Skipped:** exponential backoff headers (`Retry-After`), per-user vs. per-IP keying, burst allowance. **Risk:** IP-based limiting is bypassable with proxies; if this protects something sensitive, you'll want per-authenticated-user keys instead.
Why it failed: All assertions passed