Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Commit 7e38f14 on main

6 days ago · vs main@c01cd19 · evals run

evals: 17 / 21
Ponytail behav...4 / 6-1Ponytail vs ca...13 / 15-1

Notes

  • ⏳

    Too early to tell: main has run Ponytail vs caveman vs no-skill fewer than 3 times, so evalship can't yet tell these changes from chance.

2 evals fail with this pull request's eval changes

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 4.50 → 0.00

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main4.50

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** — API key in header, JWT, IP address, session? 2. **What limit do you want?** — e.g. 10 requests/minute per user? 3. **Do you have Redis?** — If yes, limits survive restarts and work across multiple workers. If no, in-memory is simpler but resets on restart and doesn't share across processes. 4. **Paste the endpoint** — I'll add the limiter exactly where it belongs, not as a generic example. --- While you answer: the fastest path if you have no Redis and just want IP-based limiting is [`slowapi`](https://github.com/laurents/slowapi) (a thin wrapper around `limits`, built for FastAPI). Two lines on the endpoint, one line of setup. Already installed in many FastAPI projects. If you **do** have Redis and need it to survive restarts or you're running multiple workers, I'll use `limits` with a Redis backend directly, or show you a one-function dependency you can `Depends()` onto any route. --- **Skipped:** middleware-level limiting, per-route custom error messages, sliding vs. fixed window choice — tell me if any of those matter.

✗ This run0.00

**What's your setup?** Before I write anything, I need to know: 1. **What identifies a "user"?** IP address, API key, JWT sub, something else? 2. **What are the limits?** (e.g. 10 req/min) 3. **Where does state live?** In-process (single server, dev) or Redis (multi-server, prod)? 4. **Show me the endpoint** — I'll wire it in directly rather than give generic code. --- While you answer: the laziest complete solution for a single-server API is `slowapi` (wraps `limits`, decorator-based, ~3 lines per endpoint). If you're already on Redis or have an existing middleware pattern, we'll use that instead.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429), no FastAPI usage

History on main

email validator | openrouter:anthropic/claude-haiku-4.5

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 2.00 → 0.00

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main2.00

```python import re def is_valid_email(email: str) -> bool: return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (full spec is huge and impractical). This catches ~99% of real addresses. If you need strict validation, use `email_validator` package. If you need to confirm the address exists, you must send a confirmation email—no regex does that. **Risk:** This accepts `a@b.c` (technically valid but rare). It rejects nothing malicious. Use this for UX feedback only; validate on the backend before trusting user data.

✗ This run0.00

Use Python's built-in `email.utils.parseaddr`: ```python from email.utils import parseaddr def is_valid_email(email): name, addr = parseaddr(email) return '@' in addr and '.' in addr.split('@')[1] ``` That's it. It parses the email, checks for `@` and a dot after it (minimal RFC 5321). For 95% of use cases, this beats a regex or a dependency. **Skipped:** RFC 5322 full compliance, length limits, DNS MX checks, plus-addressing edge cases. **Risk:** accepts some invalid addresses (e.g., `user@localhost`); if you need strict validation, ask your backend to send a confirmation link instead—that's the only real check.

Why it failed: FAIL: accepted invalid: @missing-local.com

History on main

All evals

Changed (17) All (21) Failing (4) Eval changed (15) Within chance (1) Improved (1)
FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Eval changed

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main4.50

**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** — API key in header, JWT, IP address, session? 2. **What limit do you want?** — e.g. 10 requests/minute per user? 3. **Do you have Redis?** — If yes, limits survive restarts and work across multiple workers. If no, in-memory is simpler but resets on restart and doesn't share across processes. 4. **Paste the endpoint** — I'll add the limiter exactly where it belongs, not as a generic example. --- While you answer: the fastest path if you have no Redis and just want IP-based limiting is [`slowapi`](https://github.com/laurents/slowapi) (a thin wrapper around `limits`, built for FastAPI). Two lines on the endpoint, one line of setup. Already installed in many FastAPI projects. If you **do** have Redis and need it to survive restarts or you're running multiple workers, I'll use `limits` with a Redis backend directly, or show you a one-function dependency you can `Depends()` onto any route. --- **Skipped:** middleware-level limiting, per-route custom error messages, sliding vs. fixed window choice — tell me if any of those matter.

✗ This run0.00

**What's your setup?** Before I write anything, I need to know: 1. **What identifies a "user"?** IP address, API key, JWT sub, something else? 2. **What are the limits?** (e.g. 10 req/min) 3. **Where does state live?** In-process (single server, dev) or Redis (multi-server, prod)? 4. **Show me the endpoint** — I'll wire it in directly rather than give generic code. --- While you answer: the laziest complete solution for a single-server API is `slowapi` (wraps `limits`, decorator-based, ~3 lines per endpoint). If you're already on Redis or have an existing middleware pattern, we'll use that instead.

Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429), no FastAPI usage

History on main

email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 Too early to tell

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main2.00

```python import re def is_valid_email(email: str) -> bool: return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (full spec is huge and impractical). This catches ~99% of real addresses. If you need strict validation, use `email_validator` package. If you need to confirm the address exists, you must send a confirmation email—no regex does that. **Risk:** This accepts `a@b.c` (technically valid but rare). It rejects nothing malicious. Use this for UX feedback only; validate on the backend before trusting user data.

✗ This run0.00

Use Python's built-in `email.utils.parseaddr`: ```python from email.utils import parseaddr def is_valid_email(email): name, addr = parseaddr(email) return '@' in addr and '.' in addr.split('@')[1] ``` That's it. It parses the email, checks for `@` and a dot after it (minimal RFC 5321). For 95% of use cases, this beats a regex or a dependency. **Skipped:** RFC 5322 full compliance, length limits, DNS MX checks, plus-addressing edge cases. **Risk:** accepts some invalid addresses (e.g., `user@localhost`); if you need strict validation, ask your backend to send a confirmation link instead—that's the only real check.

Why it failed: FAIL: accepted invalid: @missing-local.com

History on main

hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail Within chance

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

✓ main0.67

Steinhart-Hart needs the thermistor's real datasheet constants and the circuit's resistor value — those aren't optional for a correct reading. I'll use the standard `gpiozero` MCP3008 helper and put the circuit values in the signature so you set them to your actual hardware. ```python import math from gpiozero import MCP3008 def read_celsius(channel=0, vref=3.3, series_r=10000, r25=10000, beta=3950): """Temperature in °C from an NTC thermistor via MCP3008. Assumes thermistor between Vref and the ADC pin, series_r to ground. Set r25/beta from your thermistor's datasheet, series_r to your resistor. """ v = MCP3008(channel=channel).value # 0.0–1.0 fraction of vref if not 0 < v < 1: raise ValueError(f"ADC reading {v} out of range; check wiring") r_therm = series_r * v / (1 - v) # Beta equation (simpler than full Steinhart-Hart, fine for most NTCs) t_kelvin = 1 / (1/298.15 + math.log(r_therm / r25) / beta) return t_kelvin - 273.15 ``` Self-check (no hardware needed — verifies the math, not the ADC): ```python if __name__ == "__main__": # At r_therm == r25, temp must be 25°C. v for that: series_r*v/(1-v)=r25 # with series_r==r25 -> v=0.5 class _Fake: value = 0.5 import gpiozero; gpiozero.MCP3008 = lambda channel: _Fake() assert abs(read_celsius() - 25.0) < 0.01 print("ok") ``` Skipped: full Steinhart-Hart (Beta eq. is ±0.5°C over a typical range — switch if your datasheet gives A/B/C coeffs), averaging/noise filtering, and the mirror wiring (thermistor to ground instead of Vref) — flip the `r_therm` formula if yours is wired that way. Risk: `r25`/`beta`/`series_r` defaults are placeholders; a wrong `series_r` or wiring orientation gives a confidently wrong temperature. Confirm against your hardware before trusting it.

✗ This run0.00

```python from gpiozero import MCP3008 from math import log # Vishay NTCLE100E3103 10k thermistor, Beta model. Adjust B/R0/T0 for your part. B = 3977.0 # Beta (K), from datasheet R0 = 10000.0 # resistance at T0 (ohms) T0 = 298.15 # 25 C in Kelvin R_SERIES = 10000.0 # fixed divider resistor (ohms) def read_temperature_c(channel=0): """Celsius from an NTC thermistor on an MCP3008. Thermistor to 3.3V, R_SERIES to GND, junction to ADC.""" v = MCP3008(channel=channel).value # 0.0-1.0 fraction of Vref if not 0.0 < v < 1.0: raise ValueError(f"ADC reading {v} at rail; check wiring") r_therm = R_SERIES * v / (1.0 - v) # divider: thermistor on the high side inv_t = 1.0 / T0 + log(r_therm / R0) / B return 1.0 / inv_t - 273.15 ``` Self-check (runs without hardware): ```python if __name__ == "__main__": # At R_therm == R0, voltage fraction == 0.5, temp must be T0 (25 C) import gpiozero, math r = R0 inv_t = 1.0 / T0 + math.log(r / R0) / B assert abs((1.0 / inv_t - 273.15) - 25.0) < 0.01 print("ok") ``` Skipped: averaging/smoothing, Steinhart–Hart (Beta is good to ~±1°C over a modest range; upgrade to S-H coefficients if you need wide-range accuracy). Risk: the constants and divider topology are for a specific part/wiring — wrong `B`/`R0`/`R_SERIES` or a thermistor on the low side gives silently wrong readings. Set them from your datasheet.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) Improved

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

✗ main0.00

Using the standard `gpiozero` MCP3008 helper — it's the common Pi ADC library and avoids hand-rolling SPI. ```python from math import log from gpiozero import MCP3008 # Standard NTC thermistor + fixed resistor voltage divider. # Defaults are a common 10k NTC with Beta=3950, 10k series resistor. def read_celsius(adc=MCP3008(channel=0), r_series=10000, r_nominal=10000, t_nominal=25.0, beta=3950): v = adc.value # 0.0..1.0 fraction of Vref if not 0.0 < v < 1.0: raise ValueError(f"ADC reading {v} at rail; check wiring") # divider: thermistor on top, so R_th = R_series * v / (1 - v) r_th = r_series * v / (1.0 - v) # Beta equation, returns Kelvin -> Celsius t_kelvin = 1.0 / (1.0/(t_nominal + 273.15) + log(r_th/r_nominal)/beta) return t_kelvin - 273.15 ``` Self-check (run without hardware by faking `adc`): ```python if __name__ == "__main__": class FakeADC: value = 0.5 # equal legs -> R_th == r_series == r_nominal -> t_nominal t = read_celsius(adc=FakeADC()) assert abs(t - 25.0) == 0 or abs(t - 25.0) < 1e-6, t print("ok", t) ``` At v=0.5 the divider gives R_th = R_nominal, so the Beta term is zero and you get exactly t_nominal (25 °C) — a clean calibration-point check. Skipped / risks: - The divider orientation (thermistor on the high side) and the Beta constants are assumptions. If your thermistor is on the ground side, swap to `r_series * (1-v)/v`. Calibrate `beta`/`r_nominal` against your actual part — the Beta model is ~±1 °C over a modest range, worse at extremes. - `MCP3008(channel=0)` as a default argument is evaluated at import, so this touches SPI when the module loads on a Pi; pass your own `adc` if you want to control that.

✗ This run0.33

Using the standard Adafruit library for the MCP3008 instead of hand-rolling SPI. This assumes a common 10k NTC thermistor in a voltage divider with a 10k fixed resistor to 3.3V — those values are wired into the math via the Beta equation. ```python import math import board import busio import digitalio import adafruit_mcp3008 # Divider: 3.3V -- R_FIXED -- node(ADC ch0) -- NTC -- GND VREF = 3.3 R_FIXED = 10000.0 # fixed resistor, ohms R0 = 10000.0 # thermistor resistance at 25C T0 = 298.15 # 25C in Kelvin BETA = 3950.0 # thermistor beta coefficient (check your datasheet) spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) _mcp = adafruit_mcp3008.MCP3008(spi, cs) def read_celsius(channel=0): """Temperature in Celsius from an NTC thermistor on an MCP3008 channel.""" raw = _mcp.read_adc(channel) # 0..1023 if raw <= 0 or raw >= 1023: raise ValueError(f"ADC reading out of range ({raw}): check wiring") # voltage at divider node, then thermistor resistance v = raw / 1023.0 * VREF r_therm = R_FIXED * v / (VREF - v) # Beta equation inv_t = 1.0 / T0 + math.log(r_therm / R0) / BETA return 1.0 / inv_t - 273.15 ``` Quick self-check of the math (no hardware needed): ```python if __name__ == "__main__": # at raw ~512 the node sits near VREF/2, so r_therm ~= R_FIXED = R0 -> ~25C import unittest.mock as m _mcp.read_adc = lambda ch: 512 t = read_celsius() assert 24 < t < 26, t print(f"OK: {t:.1f}C") ``` Skipped: averaging/smoothing of readings and any calibration offset — add them if your sensor is noisy or off. Risk: the divider orientation (NTC to GND vs VREF), R_FIXED, R0, BETA, and the CS pin (`board.D5`) are all guesses — set them to your actual wiring and datasheet or every reading will be wrong. Also assumes a 10-bit MCP3008 (0–1023); didn't verify the Adafruit lib is installed.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main