Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Score the single-shot evals on correctness alone #5

gglucass · evalship/fixes-3 → main · 180f3d9 · 6 days ago · vs main@c01cd19 · evals run

evals: 20 / 21eval flaws: 0
Ponytail behav...5 / 6Ponytail vs ca...15 / 15+1

All evals

Changed (15) All (21) Failing (1) Eval changed (14) Now passing (1)
CSV sum | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-haiku-4.5 Eval changed

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.50

```python import pandas as pd df = pd.read_csv('sales.csv') total = df['amount'].sum() print(total) ``` That's it. If pandas is already a project dependency, use it—it's three lines and handles parsing, missing values, and type conversion. If you need the total as a return value instead of printed, wrap it in a function. **Skipped:** error handling for missing file, type validation, filtering rows. **Risk:** if 'amount' has non-numeric values, `.sum()` will skip them silently (pandas default)—add `.astype(float)` before `.sum()` if that's a problem.

✓ This PR1.00

```python import csv total = sum(float(row['amount']) for row in csv.DictReader(open('sales.csv'))) print(total) ``` **Skipped:** error handling (file missing, bad data), closing the file (context manager overkill for a script). **Risk:** if `sales.csv` has non-numeric amounts, it crashes—add `.get('amount', '0')` or a try-except if that's real data.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-opus-4.8 Eval changed

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.50

```python import csv with open("sales.csv", newline="") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(total) ``` Skipped: error handling for missing file/bad values (not asked). Risk: assumes a header row named `amount` and that every value parses as a number—one bad cell raises.

✓ This PR1.00

```python import csv with open("sales.csv", newline="") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(total) ``` Skipped: no error handling for missing file/column or non-numeric values (add if the CSV is untrusted). Didn't run it—no sample file.

Why it failed: All assertions passed

History on main

CSV sum | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / CSV sum | openrouter:anthropic/claude-sonnet-4.6 Eval changed

Input

{"task":"Write Python code that reads sales.csv and sums the 'amount' column."}

✓ main2.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` Skipped: error handling for missing file or bad values. Risk: crashes on empty `amount` cells — replace `float(...)` with `float(row["amount"] or 0)` if needed.

✓ This PR1.00

```python import csv with open("sales.csv") as f: print(sum(float(row["amount"]) for row in csv.DictReader(f))) ``` **Skipped:** error handling for missing file or bad values. **Risk:** crashes on blank or non-numeric `amount` cells.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-haiku-4.5 Eval changed

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main9.50

# Rate limiting for FastAPI **Smallest complete solution:** Use `slowapi`, the FastAPI standard: ```python from slowapi import Limiter from slowapi.util import get_remote_address limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): return {"data": "value"} ``` **Install:** ```bash pip install slowapi ``` **Add to exception handlers** (one time, in your main app file): ```python from slowapi.errors import RateLimitExceeded from fastapi.responses import JSONResponse @app.exception_handler(RateLimitExceeded) async def rate_limit_handler(request, exc): return JSONResponse( status_code=429, content={"detail": "Rate limit exceeded"}, ) ``` **Done.** Apply `@limiter.limit("N/timeunit")` to any endpoint. Timeunits: `second`, `minute`, `hour`, `day`. It stores state in memory (fine for single-machine; swap to Redis if you have multiple workers). --- **Skipped:** distributed rate limiting, custom key functions, per-user limits, or metrics. **Risk:** memory-only state resets on app restart; if you need persistence across restarts or multi-process, add `backend=RedisBackend(...)` to the Limiter.

✓ This PR1.00

# Minimal rate limiting for FastAPI **Smallest complete solution:** ```python from slowapi import Limiter from slowapi.util import get_remote_address from fastapi import FastAPI limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): return {"status": "ok"} ``` **Install:** `pip install slowapi` That's it. The decorator handles rejection (HTTP 429) and headers automatically. Slowapi is already in most FastAPI projects; if not, it's 50 lines of code vs. the dependency, so the dependency wins. **If you need more endpoints**, just stack the decorator with different limits. **To customize error response**, wrap the limiter in your error handler. --- **Skipped:** in-memory vs. Redis (in-memory works unless you have multiple workers; then use Redis). Distributed rate limiting setup. Custom key functions for user IDs instead of IP (trivial to add later: `key_func=lambda r: r.headers.get("user-id")`). **Risk:** Respects X-Forwarded-For headers—safe behind a proxy, but if you're not behind one and users can spoof headers, fall back to raw client IP.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-opus-4.8 Eval changed

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✓ main5.50

Use `slowapi` — it's the standard FastAPI rate limiter, a few lines. Install: `pip install slowapi` ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") async def your_endpoint(request: Request): # `request` param is required by slowapi ... ``` Change `5/minute` to whatever "spam" means for you. Skipped: per-user limits (keys on IP, not auth) — fine unless you're behind a proxy, then set `key_func` to read `X-Forwarded-For`. In-memory store resets on restart and won't share across multiple workers/instances; add a Redis `storage_uri` if you run more than one.

✓ This PR1.00

Use `slowapi` (the standard FastAPI rate-limit library, wraps `limits`). ~4 lines of setup plus a decorator. ```python # app.py from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded from fastapi import FastAPI, Request limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("5/minute") # tune the number def your_endpoint(request: Request): # `request` param is required by slowapi ... ``` Install: `pip install slowapi`. Two things to know: - **`get_remote_address` uses the socket IP.** Behind a proxy/load balancer that's the proxy, not the user — everyone shares one limit. If you're behind one, set `key_func` to read `X-Forwarded-For` (and make sure the proxy sets it). - **Default storage is in-memory**, so limits reset on restart and aren't shared across workers/replicas. For multiple workers, point it at Redis: `Limiter(key_func=..., storage_uri="redis://...")`. Skipped: didn't add Redis or proxy handling since I don't know your deploy setup — pick based on the two notes above. Didn't write a test; rate-limit behavior is slowapi's to guarantee, not yours.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-haiku-4.5 Eval changed

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // Only this fires, 300ms later ``` --- **Skipped:** cancellation method, trailing/leading edge options, return value from fn, max-wait limit. Add any if you need them. **Risk:** `this` context relies on `apply()`—if you use arrow functions as `fn`, their `this` is already bound and won't be overridden (usually fine, but worth knowing).

✓ This PR1.00

```javascript function debounce(fn, delay) { let timeoutId; return function(...args) { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn(...args), delay); }; } ``` **Usage:** ```javascript const search = debounce((query) => console.log('Searching:', query), 300); search('a'); search('ab'); search('abc'); // Only this runs after 300ms ``` --- **Skipped:** cancel method, leading/trailing options, immediate flag, result promises, this binding preservation (your args already work). These are "nice to have" until you need them. **Risk:** If `fn` throws, the error surfaces on the delayed call (not in the caller). If you need to catch it, wrap the `fn` call: `setTimeout(() => { try { fn(...args); } catch(e) { /* handle */ } }, delay)`.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-opus-4.8 Eval changed

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js let calls = 0; const d = debounce(() => calls++, 50); d(); d(); d(); setTimeout(() => console.assert(calls === 1, `expected 1, got ${calls}`), 100); ``` Skipped: leading-edge, cancel/flush methods, and a return value (debounced calls are async, so there's nothing to return). Say the word if you need any of those.

✓ This PR1.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Preserves `this` and args so it works on methods and event handlers too. Skipped: cancel/flush methods and leading-edge option (not asked). No test — logic is trivial and timing-dependent.

Why it failed: All assertions passed

History on main

JS debounce | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / JS debounce | openrouter:anthropic/claude-sonnet-4.6 Eval changed

Input

{"task":"Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call."}

✓ main6.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` **Self-check** (runs once in Node or a browser console): ```js // quick smoke-test — remove in production let calls = 0; const inc = debounce(() => calls++, 50); inc(); inc(); inc(); setTimeout(() => console.assert(calls === 1, "debounce failed"), 100); ``` **Skipped:** leading-edge ("immediate") option, cancel/flush methods — not asked for. **Risk:** `this` binding relies on normal `function` syntax; arrow-function callers won't need it, but class method callers will — works correctly either way.

✓ This PR1.00

```js function debounce(fn, delay) { let timer; return function (...args) { clearTimeout(timer); timer = setTimeout(() => fn.apply(this, args), delay); }; } ``` Self-check: ```js let count = 0; const inc = debounce(() => count++, 50); inc(); inc(); inc(); setTimeout(() => console.assert(count === 1, "should fire once"), 100); ``` **Skipped:** leading-edge firing, cancel/flush methods — add when needed. **Risk:** `timer` is shared across all calls; that is the intended behaviour, but means one returned function = one independent debounce scope.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-haiku-4.5 Eval changed

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main7.00

# Countdown Timer ```jsx export function CountdownTimer({ seconds, onComplete }) { const [timeLeft, setTimeLeft] = useState(seconds); useEffect(() => { if (timeLeft <= 0) { onComplete?.(); return; } const timer = setInterval(() => setTimeLeft(t => t - 1), 1000); return () => clearInterval(timer); }, [timeLeft, onComplete]); return <div>{timeLeft}s</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={60} onComplete={() => console.log('done')} /> ``` --- **Skipped:** reset button, pause, display format (mm:ss), animations, styling, aria labels, effect cleanup on unmount edge case. **Risk:** `onComplete` in the dependency array could fire multiple times if you pass a new function every render—wrap it in `useCallback` if that's your pattern. If `seconds` changes mid-countdown, it restarts.

✓ This PR1.00

# Countdown Timer ```jsx export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = React.useState(seconds); React.useEffect(() => { if (remaining <= 0) return; const timer = setTimeout(() => setRemaining(remaining - 1), 1000); return () => clearTimeout(timer); }, [remaining]); return <div>{remaining}</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={10} /> ``` --- **What I skipped:** pause/resume, restart button, callbacks on zero, formatting (mm:ss), animations, accessibility announcements. Add only if you need them. **Risk:** If `seconds` prop changes mid-countdown, it resets. If that's wrong, add a key or a separate "start" trigger.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-opus-4.8 Eval changed

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main7.00

```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { setRemaining(seconds); if (seconds <= 0) return; const id = setInterval(() => { setRemaining((r) => (r <= 1 ? (clearInterval(id), 0) : r - 1)); }, 1000); return () => clearInterval(id); }, [seconds]); return <span>{remaining}</span>; } ``` Skipped (nobody asked): pause/reset, mm:ss formatting, onComplete callback — add if you need them. Risk: `setInterval` drifts over long runs; fine for a display timer, not for anything billed by the second.

✓ This PR1.00

```jsx import { useState, useEffect } from "react"; export function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); useEffect(() => { setRemaining(seconds); const id = setInterval(() => { setRemaining((s) => (s <= 0 ? 0 : s - 1)); }, 1000); return () => clearInterval(id); }, [seconds]); return <span>{remaining}</span>; } ``` Usage: `<CountdownTimer seconds={60} />` Skipped (nobody asked): start/pause controls, onComplete callback, mm:ss formatting, styling. Say the word and I'll add any of them. Risk: `setInterval` drifts over long runs; fine for a display timer, not for timekeeping. It keeps ticking at 0 (harmless, just re-renders the same value) — one `clearInterval` when it hits 0 would stop that if you care.

Why it failed: All assertions passed

History on main

React countdown | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / React countdown | openrouter:anthropic/claude-sonnet-4.6 Eval changed

Input

{"task":"Build me a countdown timer component in React that counts down from a given number of seconds."}

✓ main13.50

Here's the minimal component: ```jsx // CountdownTimer.jsx import { useState, useEffect, useRef } from "react"; export default function CountdownTimer({ seconds, onComplete }) { const [remaining, setRemaining] = useState(seconds); const intervalRef = useRef(null); useEffect(() => { if (remaining <= 0) { onComplete?.(); return; } intervalRef.current = setInterval(() => { setRemaining((r) => { if (r <= 1) { clearInterval(intervalRef.current); onComplete?.(); return 0; } return r - 1; }); }, 1000); return () => clearInterval(intervalRef.current); }, []); // run once on mount const m = Math.floor(remaining / 60).toString().padStart(2, "0"); const s = (remaining % 60).toString().padStart(2, "0"); return <div aria-live="polite">{m}:{s}</div>; } ``` **Usage:** ```jsx <CountdownTimer seconds={90} onComplete={() => alert("Done!")} /> ``` --- **Skipped:** pause/resume/reset controls, resetting when `seconds` prop changes after mount, and any visual styling — add those if you need them. **Risk:** the interval drifts slightly over long durations (~1 s per few minutes); use `Date`-based logic if precision matters.

✓ This PR1.00

Here's the minimal working component: ```jsx import { useState, useEffect, useRef } from "react"; export default function CountdownTimer({ seconds }) { const [remaining, setRemaining] = useState(seconds); const intervalRef = useRef(null); useEffect(() => { if (remaining <= 0) return; intervalRef.current = setInterval(() => { setRemaining((r) => { if (r <= 1) clearInterval(intervalRef.current); return Math.max(0, r - 1); }); }, 1000); return () => clearInterval(intervalRef.current); }, []); // run once on mount const mm = String(Math.floor(remaining / 60)).padStart(2, "0"); const ss = String(remaining % 60).padStart(2, "0"); return <output>{mm}:{ss}</output>; } ``` **Usage:** ```jsx <CountdownTimer seconds={90} /> ``` Displays `01:30` and counts to `00:00`. --- **Skipped:** pause/reset controls, `onComplete` callback, styling, syncing if `seconds` prop changes after mount — add any of these only when needed. **Risk:** timer can drift by up to ~1 s/minute on a busy main thread; fine for most uses, not for precision countdowns.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 Eval changed

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main2.00

```python import re def is_valid_email(email: str) -> bool: return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (full spec is huge and impractical). This catches ~99% of real addresses. If you need strict validation, use `email_validator` package. If you need to confirm the address exists, you must send a confirmation email—no regex does that. **Risk:** This accepts `a@b.c` (technically valid but rare). It rejects nothing malicious. Use this for UX feedback only; validate on the backend before trusting user data.

✓ This PR1.00

```python import re def validate_email(email): return bool(re.match(r'^[^@]+@[^@]+\.[^@]+$', email)) ``` **What I skipped:** RFC 5322 compliance (26 KB spec), DNS MX lookups, disposable domain lists, internationalized domains. The regex above catches 99% of real-world cases and stops obvious junk. **Risk:** This regex is permissive—it accepts `a@b.c` and won't catch typos. If you need to actually *reach* the user, send a confirmation email instead of validating harder here. If this is for a sign-up form, that's your real gate.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-opus-4.8Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-opus-4.8 Eval changed

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main4.50

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.fullmatch(email)) ``` Quick self-check: ```python assert is_valid_email("a@b.co") assert not is_valid_email("a@b") # no TLD assert not is_valid_email("a b@c.co") # space assert not is_valid_email("a@@b.co") # double @ ``` Skipped: full RFC 5322 (quoted locals, IP literals, comments) — nobody needs it and the regex gets monstrous. Risk: this is a format check, not a deliverability check; the only way to know an address works is to send to it.

✓ This PR1.00

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(email: str) -> bool: return _EMAIL_RE.fullmatch(email) is not None ``` ```python assert is_valid_email("a@b.co") assert not is_valid_email("a@b") assert not is_valid_email("a b@c.co") assert not is_valid_email("@b.co") ``` Skipped: no full RFC 5322 parsing (quoted locals, IP literals) — nobody asked, and even the "correct" regex rejects valid addresses in practice. Risk: this accepts syntactically-odd-but-real addresses and rejects a few exotic valid ones; the only sure check is sending a confirmation email, so use that if correctness matters.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-sonnet-4.6 Eval changed

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main3.50

```python import re _EMAIL_RE = re.compile(r"[^@\s]+@[^@\s]+\.[^@\s]+") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.fullmatch(email)) ``` **Usage** ```python is_valid_email("user@example.com") # True is_valid_email("bad@") # False ``` --- Skipped: RFC 5322 edge cases (quoted locals, IP literals, comments) — real-world emails don't use them. Risk: rejects technically valid but exotic addresses like `"user name"@example.com`; use a library like [`email-validator`](https://pypi.org/project/email-validator/) if you need strict RFC compliance or MX-record checking.

✓ This PR1.00

```python import re _EMAIL_RE = re.compile(r"[^@]+@[^@]+\.[^@]+") def is_valid_email(email: str) -> bool: return bool(_EMAIL_RE.match(email)) ``` **Skipped:** RFC 5322 edge cases (quoted locals, IP literals, long TLDs). **Risk:** if you're gatekeeping real user data, use `email-validator` on PyPI instead.

Why it failed: All assertions passed

History on main

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Now passing

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main4.50

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** — API key in header, JWT, IP address, session? 2. **What limit do you want?** — e.g. 10 requests/minute per user? 3. **Do you have Redis?** — If yes, limits survive restarts and work across multiple workers. If no, in-memory is simpler but resets on restart and doesn't share across processes. 4. **Paste the endpoint** — I'll add the limiter exactly where it belongs, not as a generic example. --- While you answer: the fastest path if you have no Redis and just want IP-based limiting is [`slowapi`](https://github.com/laurents/slowapi) (a thin wrapper around `limits`, built for FastAPI). Two lines on the endpoint, one line of setup. Already installed in many FastAPI projects. If you **do** have Redis and need it to survive restarts or you're running multiple workers, I'll use `limits` with a Redis backend directly, or show you a one-function dependency you can `Depends()` onto any route. --- **Skipped:** middleware-level limiting, per-route custom error messages, sliding vs. fixed window choice — tell me if any of those matter.

✓ This PR1.00

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** (API key in header, JWT, IP address, session cookie?) 2. **What's the limit?** (e.g. "10 requests per minute") 3. **Do you have Redis?** (or is this a single-process app where in-memory is fine?) --- That said, here's the **smallest complete solution for the common case** — IP-based, in-memory, no new dependencies: **`slowapi` is already the FastAPI-idiomatic choice** if you want something maintained. But if you want zero deps: ```python import time from collections import defaultdict from fastapi import Request, HTTPException # shortcut: single-process only, resets on restart; switch to Redis when you scale out _requests: dict[str, list[float]] = defaultdict(list) LIMIT = 10 # requests WINDOW = 60.0 # seconds def rate_limit(request: Request, limit: int = LIMIT, window: float = WINDOW): key = request.client.host now = time.monotonic() hits = _requests[key] # drop timestamps outside the window _requests[key] = [t for t in hits if now - t < window] if len(_requests[key]) >= limit: raise HTTPException(status_code=429, detail="Too many requests") _requests[key].append(now) # usage from fastapi import Depends @app.get("/your-endpoint") def your_endpoint(request: Request, _=Depends(rate_limit)): ... ``` **If you have slowapi installed** (or are open to it — it's the standard pick): ```python # pip install slowapi from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("10/minute") async def your_endpoint(request: Request): ... ``` --- **Skipped:** per-user limits, Redis backend, custom 429 response body, rate-limit headers (`X-RateLimit-*`). **Risk:** IP-based limiting breaks behind a shared NAT or proxy — all users share one bucket. Tell me your auth setup if that matters.

Why it failed: All assertions passed

History on main