Score the single-shot evals on correctness alone #5
gglucass · evalship/fixes-3 → main · 180f3d9 ·
6 days ago · vs main@c01cd19
· evals run
All evals
FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Now passing
Input
{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}
✗ main4.50
**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** — API key in header, JWT, IP address, session? 2. **What limit do you want?** — e.g. 10 requests/minute per user? 3. **Do you have Redis?** — If yes, limits survive restarts and work across multiple workers. If no, in-memory is simpler but resets on restart and doesn't share across processes. 4. **Paste the endpoint** — I'll add the limiter exactly where it belongs, not as a generic example. --- While you answer: the fastest path if you have no Redis and just want IP-based limiting is [`slowapi`](https://github.com/laurents/slowapi) (a thin wrapper around `limits`, built for FastAPI). Two lines on the endpoint, one line of setup. Already installed in many FastAPI projects. If you **do** have Redis and need it to survive restarts or you're running multiple workers, I'll use `limits` with a Redis backend directly, or show you a one-function dependency you can `Depends()` onto any route. --- **Skipped:** middleware-level limiting, per-route custom error messages, sliding vs. fixed window choice — tell me if any of those matter.
✓ This PR1.00
**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** (API key in header, JWT, IP address, session cookie?) 2. **What's the limit?** (e.g. "10 requests per minute") 3. **Do you have Redis?** (or is this a single-process app where in-memory is fine?) --- That said, here's the **smallest complete solution for the common case** — IP-based, in-memory, no new dependencies: **`slowapi` is already the FastAPI-idiomatic choice** if you want something maintained. But if you want zero deps: ```python import time from collections import defaultdict from fastapi import Request, HTTPException # shortcut: single-process only, resets on restart; switch to Redis when you scale out _requests: dict[str, list[float]] = defaultdict(list) LIMIT = 10 # requests WINDOW = 60.0 # seconds def rate_limit(request: Request, limit: int = LIMIT, window: float = WINDOW): key = request.client.host now = time.monotonic() hits = _requests[key] # drop timestamps outside the window _requests[key] = [t for t in hits if now - t < window] if len(_requests[key]) >= limit: raise HTTPException(status_code=429, detail="Too many requests") _requests[key].append(now) # usage from fastapi import Depends @app.get("/your-endpoint") def your_endpoint(request: Request, _=Depends(rate_limit)): ... ``` **If you have slowapi installed** (or are open to it — it's the standard pick): ```python # pip install slowapi from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): ... ``` --- **Skipped:** per-user limits, Redis backend, custom 429 response body, rate-limit headers (`X-RateLimit-*`). **Risk:** IP-based limiting breaks behind a shared NAT or proxy — all users share one bucket. Tell me your auth setup if that matters.
Why it failed: All assertions passed