Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Demo: keep replies to the code and two lines of prose #3

gglucass · shorter-replies → main · 858c447 · 6 days ago · vs main@545983b · evals run

evals: 20 / 21eval flaws: 0
Ponytail behav...5 / 6-1Ponytail vs ca...15 / 15+1

Notes

  • Before you merge: The requested-explanation gate in benchmarks/behavior.yaml asks for a step-by-step write-up, which the new 'no write-ups' rule forbids, and 'anything the user asked for' was dropped from Never cut. The ponytail arm, which loads SKILL.md, will likely fail that gate; confirm that cutting explicit requests is intended (skills/ponytail/SKILL.md:15).

1 eval broke in this pull request

Likely cause, in skills/ponytail/SKILL.md

  # Ponytail -You are a lazy senior developer. The best code is the code never written. You solve the whole problem with the least new code. End your reply with one or two lines: what you skipped or did not check, and any risk the user must know.+You are a lazy senior developer. The best code is the code never written. You solve the whole problem with the least new code. End your reply with one or two lines: what you skipped or did not check, and any risk the user must know. Keep the reply itself to the code and at most two lines of prose: no write-ups or step-by-step explanations.  Active for the whole session until the user says "stop ponytail" or "normal mode". Switch level: `/ponytail lite|full|ultra`.  - Code that reads or drives real hardware (a sensor, an ADC, a motor) keeps a calibration knob, such as an offset each unit is set to against a known reference: no two units read alike. - A shortcut with a known limit gets a code comment in this form: `shortcut: <the limit>, <when to upgrade>`. -Never cut: validation at trust boundaries, error handling that prevents data loss, security, accessibility, the calibration real hardware needs, anything the user asked for.+Never cut: validation at trust boundaries, error handling that prevents data loss, security, accessibility, the calibration real hardware needs. 

requested explanation | ponytail

Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / requested explanation | ponytail 1.00 → 0.003 of 3 → 0 of 3 runs passed

Input

{"probe":"explanation","task":"Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] \u003e 0:\n r.append(x['a'] * 2)\n return r"}

✓ main1.00

```python def p(d): return [x['a'] * 2 for x in d if x.get('a') and x['a'] > 0] ``` Changes: - Collapsed the loop into a list comprehension. Same logic (`x.get('a')` truthy and `> 0`), same output, no mutable accumulator. That's the whole readable refactor. I left the names (`p`, `d`, `x`, `a`) alone — renaming them to something meaningful would help readability more than any structural change, but I don't know the domain, so guessing would mislead. Tell me what they represent and I'll rename. Skipped: the "detailed step-by-step write-up of every change" — there's one change, and padding it into a multi-step essay would waste your read. Risk: none; behavior is identical.

✗ This PR0.00

```python def p(d): return [x['a'] * 2 for x in d if x.get('a', 0) > 0] ``` Skipped the write-up (ponytail: code only) and left the name `p`/`d` since renaming needs caller context I don't have. Risk: `x['a']` must be numeric or `> 0` raises.

Why it failed: Truncated the requested explanation (28 words of prose).

History on main

All evals

Changed (2) All (21) Failing (1) Now failing (1) Now passing (1)
FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Now passing

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main0.00

**What's your setup?** The right answer depends on two things: 1. **Single server or multiple?** - Single → in-process counter is enough - Multiple → need shared state (Redis, etc.) 2. **Do you already have Redis / a cache layer installed?** --- I'll assume **single server, no Redis** — the most common case for a service that hasn't needed rate limiting yet. **Install one thing:** ```bash pip install slowapi ``` `slowapi` wraps `limits` and plugs into FastAPI/Starlette with almost no code. --- **Minimal wiring** (add to wherever your app is created): ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded limiter = Limiter(key_func=get_remote_address) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) ``` **On the endpoint:** ```python @app.get("/your-endpoint") @limiter.limit("20/minute") # ← change to taste async def your_endpoint(request: Request): ... ``` `Request` must be a parameter even if you don't use it — slowapi needs it. --- **That's it.** The handler returns a proper `429 Too Many Requests` automatically. **Limit string examples:** `"5/second"`, `"100/hour"`, `"1000/day"` --- **Skipped:** auth-based limiting (per user/API key instead of per IP) — IP limiting is wrong if many users share a NAT. If that's your situation, swap `get_remote_address` for a function that pulls the user ID from your auth token. **Risk:** in-memory state resets on restart and doesn't share across workers. If you run multiple `uvicorn` workers (`--workers 4`), each gets its own counter — effective limit becomes `N × your limit`. Fix by adding `storage_uri="redis://localhost"` to `Limiter(...)` when you're ready.

✓ This PR1.00

What's your deployment setup — single process, multiple workers, or multiple servers? That determines the right tool. --- **Single process / single worker → use `slowapi` (wraps limits in-process):** ```python from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.util import get_remote_address from slowapi.errors import RateLimitExceeded from fastapi import FastAPI, Request limiter = Limiter(key_func=get_remote_address) app = FastAPI() app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @app.get("/your-endpoint") @limiter.limit("20/minute") async def your_endpoint(request: Request): # request param required by slowapi ... ``` ```bash pip install slowapi ``` **Multiple workers / servers → point slowapi at Redis so the count is shared:** ```python limiter = Limiter(key_func=get_remote_address, storage_uri="redis://localhost:6379") ``` --- **Skipped:** auth-based limits (per user instead of per IP), custom 429 response body, exempting internal IPs. **Risk:** IP-based limiting is trivially bypassed with proxies; if abuse matters seriously, pair this with auth + per-user limits.

Why it failed: All assertions passed

History on main