Commit c01cd19 on main
6 days ago · vs main@1e356df · evals run
Notes
-
New, failing here:
FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6fails,hardware calibration | ponytail (AGENTS.md)fails (2 of the 21 evals with no result onmain). -
ℹ️
2 evals that ran on
main@1e356dfdidn't run here, so they're left out of the comparison. -
ℹ️
None of the 21 evals here have a result on
mainto compare with; they are new, didn't run there in the last 10 runs, or couldn't run.
All evals
hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) New
Input
{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}
Output
Using the standard `gpiozero` MCP3008 helper — it's the common Pi ADC library and avoids hand-rolling SPI. ```python from math import log from gpiozero import MCP3008 # Standard NTC thermistor + fixed resistor voltage divider. # Defaults are a common 10k NTC with Beta=3950, 10k series resistor. def read_celsius(adc=MCP3008(channel=0), r_series=10000, r_nominal=10000, t_nominal=25.0, beta=3950): v = adc.value # 0.0..1.0 fraction of Vref if not 0.0 < v < 1.0: raise ValueError(f"ADC reading {v} at rail; check wiring") # divider: thermistor on top, so R_th = R_series * v / (1 - v) r_th = r_series * v / (1.0 - v) # Beta equation, returns Kelvin -> Celsius t_kelvin = 1.0 / (1.0/(t_nominal + 273.15) + log(r_th/r_nominal)/beta) return t_kelvin - 273.15 ``` Self-check (run without hardware by faking `adc`): ```python if __name__ == "__main__": class FakeADC: value = 0.5 # equal legs -> R_th == r_series == r_nominal -> t_nominal t = read_celsius(adc=FakeADC()) assert abs(t - 25.0) == 0 or abs(t - 25.0) < 1e-6, t print("ok", t) ``` At v=0.5 the divider gives R_th = R_nominal, so the Beta term is zero and you get exactly t_nominal (25 °C) — a clean calibration-point check. Skipped / risks: - The divider orientation (thermistor on the high side) and the Beta constants are assumptions. If your thermistor is on the ground side, swap to `r_series * (1-v)/v`. Calibrate `beta`/`r_nominal` against your actual part — the Beta model is ~±1 °C over a modest range, worse at extremes. - `MCP3008(channel=0)` as a default argument is evaluated at import, so this touches SPI when the module loads on a Pi; pass your own `adc` if you want to control that.
Why it failed: Treats the hardware as ideal; no calibration knob.
FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 New
Input
{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}
Output
**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** — API key in header, JWT, IP address, session? 2. **What limit do you want?** — e.g. 10 requests/minute per user? 3. **Do you have Redis?** — If yes, limits survive restarts and work across multiple workers. If no, in-memory is simpler but resets on restart and doesn't share across processes. 4. **Paste the endpoint** — I'll add the limiter exactly where it belongs, not as a generic example. --- While you answer: the fastest path if you have no Redis and just want IP-based limiting is [`slowapi`](https://github.com/laurents/slowapi) (a thin wrapper around `limits`, built for FastAPI). Two lines on the endpoint, one line of setup. Already installed in many FastAPI projects. If you **do** have Redis and need it to survive restarts or you're running multiple workers, I'll use `limits` with a Redis backend directly, or show you a one-function dependency you can `Depends()` onto any route. --- **Skipped:** middleware-level limiting, per-route custom error messages, sliding vs. fixed window choice — tell me if any of those matter.
Why it failed: Missing: no rate limiting (a limiter, or requests counted over time and rejected with a 429)