Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Fix the eval failing on main: hardware calibration #4

gglucass · evalship/failing-1 → main · cd02498 · 6 days ago · vs main@7e38f14 · evals run

evals: 21 / 21eval flaws: 0
Ponytail behav...6 / 6+2Ponytail vs ca...15 / 15+2

0 eval flaws in this pull request

Untested in hooks/ponytail-instructions.js, dismissed by gglucass as later: the fallback repeats AGENTS.md's rules and runs only when SKILL.md can't be read; a test that it matches AGENTS.md would cover it (hooks/ponytail-instructions.js:44).

hooks/ponytail-instructions.js#getFallbackInstructions has no eval.

All evals

Changed (4) All (21) Now passing (4)
hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail Now passing

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

✗ main0.00

```python from gpiozero import MCP3008 from math import log # Vishay NTCLE100E3103 10k thermistor, Beta model. Adjust B/R0/T0 for your part. B = 3977.0 # Beta (K), from datasheet R0 = 10000.0 # resistance at T0 (ohms) T0 = 298.15 # 25 C in Kelvin R_SERIES = 10000.0 # fixed divider resistor (ohms) def read_temperature_c(channel=0): """Celsius from an NTC thermistor on an MCP3008. Thermistor to 3.3V, R_SERIES to GND, junction to ADC.""" v = MCP3008(channel=channel).value # 0.0-1.0 fraction of Vref if not 0.0 < v < 1.0: raise ValueError(f"ADC reading {v} at rail; check wiring") r_therm = R_SERIES * v / (1.0 - v) # divider: thermistor on the high side inv_t = 1.0 / T0 + log(r_therm / R0) / B return 1.0 / inv_t - 273.15 ``` Self-check (runs without hardware): ```python if __name__ == "__main__": # At R_therm == R0, voltage fraction == 0.5, temp must be T0 (25 C) import gpiozero, math r = R0 inv_t = 1.0 / T0 + math.log(r / R0) / B assert abs((1.0 / inv_t - 273.15) - 25.0) < 0.01 print("ok") ``` Skipped: averaging/smoothing, Steinhart–Hart (Beta is good to ~±1°C over a modest range; upgrade to S-H coefficients if you need wide-range accuracy). Risk: the constants and divider topology are for a specific part/wiring — wrong `B`/`R0`/`R_SERIES` or a thermistor on the low side gives silently wrong readings. Set them from your datasheet.

✓ This PR1.00

```python import math from gpiozero import MCP3008 # Calibrate per unit: Steinhart–Hart B-coefficient and divider resistor. def read_celsius(channel=0, series_r=10000, nominal_r=10000, nominal_t=25, beta=3950, offset=0.0): adc = MCP3008(channel=channel) v = adc.value # 0.0..1.0 fraction of Vref if not 0 < v < 1: raise ValueError(f"ADC reading {v} out of range (open/short thermistor?)") r = series_r * v / (1 - v) # thermistor resistance, thermistor on high side inv_t = 1 / (nominal_t + 273.15) + math.log(r / nominal_r) / beta return 1 / inv_t - 273.15 + offset ``` Self-check: ```python if __name__ == "__main__": # At nominal_r the formula must return nominal_t. import unittest.mock as m with m.patch("gpiozero.MCP3008") as fake: fake.return_value.value = 0.5 # r == series_r == nominal_r assert abs(read_celsius() - 25) < 0.01 print("ok") ``` Skipped: averaging/smoothing and a repeat loop — add if your reading is noisy. Assumed the thermistor sits on the high side of the divider and `MCP3008.value` is the Vref fraction (true for gpiozero); flip the `r =` line if yours is wired low-side. `beta`/`nominal_r` are placeholders — set them from your thermistor's datasheet and trim `offset` against a known reference, or the reading is meaningless.

History on main

hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) Now passing

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

✗ main0.33

Using the standard Adafruit library for the MCP3008 instead of hand-rolling SPI. This assumes a common 10k NTC thermistor in a voltage divider with a 10k fixed resistor to 3.3V — those values are wired into the math via the Beta equation. ```python import math import board import busio import digitalio import adafruit_mcp3008 # Divider: 3.3V -- R_FIXED -- node(ADC ch0) -- NTC -- GND VREF = 3.3 R_FIXED = 10000.0 # fixed resistor, ohms R0 = 10000.0 # thermistor resistance at 25C T0 = 298.15 # 25C in Kelvin BETA = 3950.0 # thermistor beta coefficient (check your datasheet) spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) _mcp = adafruit_mcp3008.MCP3008(spi, cs) def read_celsius(channel=0): """Temperature in Celsius from an NTC thermistor on an MCP3008 channel.""" raw = _mcp.read_adc(channel) # 0..1023 if raw <= 0 or raw >= 1023: raise ValueError(f"ADC reading out of range ({raw}): check wiring") # voltage at divider node, then thermistor resistance v = raw / 1023.0 * VREF r_therm = R_FIXED * v / (VREF - v) # Beta equation inv_t = 1.0 / T0 + math.log(r_therm / R0) / BETA return 1.0 / inv_t - 273.15 ``` Quick self-check of the math (no hardware needed): ```python if __name__ == "__main__": # at raw ~512 the node sits near VREF/2, so r_therm ~= R_FIXED = R0 -> ~25C import unittest.mock as m _mcp.read_adc = lambda ch: 512 t = read_celsius() assert 24 < t < 26, t print(f"OK: {t:.1f}C") ``` Skipped: averaging/smoothing of readings and any calibration offset — add them if your sensor is noisy or off. Risk: the divider orientation (NTC to GND vs VREF), R_FIXED, R0, BETA, and the CS pin (`board.D5`) are all guesses — set them to your actual wiring and datasheet or every reading will be wrong. Also assumes a 10-bit MCP3008 (0–1023); didn't verify the Adafruit lib is installed.

✓ This PR1.00

Using the existing `adafruit-circuitpython-mcp3xxx` library (the standard for this chip) rather than bit-banging SPI. ```python import math import board, busio, digitalio import adafruit_mcp3xxx.mcp3008 as MCP from adafruit_mcp3xxx.analog_in import AnalogIn # Beta-equation constants for a common 10k NTC thermistor. Adjust to your part. SERIES_R = 10_000.0 # fixed divider resistor, ohms R0 = 10_000.0 # thermistor resistance at T0 T0 = 298.15 # 25 C in kelvin BETA = 3950.0 CAL_OFFSET_C = 0.0 # per-unit calibration against a known reference def read_temp_c(cal_offset_c=CAL_OFFSET_C): spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) chan = AnalogIn(MCP.MCP3008(spi, cs), MCP.P0) # voltage divider: thermistor on top, SERIES_R to ground v_ratio = chan.value / 65535.0 # 16-bit normalized reading if v_ratio <= 0.0 or v_ratio >= 1.0: raise ValueError(f"ADC reading out of range: {chan.value}") r_therm = SERIES_R * (1.0 / v_ratio - 1.0) inv_t = 1.0 / T0 + math.log(r_therm / R0) / BETA return 1.0 / inv_t - 273.15 + cal_offset_c ``` Self-check (runs without hardware, exercises only the math): ```python if __name__ == "__main__" and False: # flip to True for the pure-math check pass def _selfcheck(): import adafruit_mcp3xxx.mcp3008 # noqa # at 25C the divider sits at mid-scale -> expect ~25C v_ratio = 0.5 r = SERIES_R * (1.0 / v_ratio - 1.0) t = 1.0 / (1.0 / T0 + math.log(r / R0) / BETA) - 273.15 assert abs(t - 25.0) < 0.1, t ``` I skipped the actual hardware read in the self-check (can't run SPI off-Pi) and only verified the Beta math. Risks you must know: the constants (`BETA`, `R0`, `SERIES_R`) and the CS pin (`D5`) are for a typical setup — set them to your thermistor's datasheet and your wiring, and set `CAL_OFFSET_C` per unit against a reference. The divider topology assumed is thermistor-to-3.3V, `SERIES_R`-to-ground; flip the `r_therm` formula if yours is reversed.

History on main

FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | openrouter:anthropic/claude-sonnet-4.6 Now passing

Input

{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}

✗ main0.00

**What's your setup?** Before I write anything, I need to know: 1. **What identifies a "user"?** IP address, API key, JWT sub, something else? 2. **What are the limits?** (e.g. 10 req/min) 3. **Where does state live?** In-process (single server, dev) or Redis (multi-server, prod)? 4. **Show me the endpoint** — I'll wire it in directly rather than give generic code. --- While you answer: the laziest complete solution for a single-server API is `slowapi` (wraps `limits`, decorator-based, ~3 lines per endpoint). If you're already on Redis or have an existing middleware pattern, we'll use that instead.

✓ This PR1.00

**What's already in your project?** I don't have your code, so I'll use the most common FastAPI setup. Adjust the import path if you already have a Redis client or a different cache. --- ## Simplest complete solution: in-process sliding window No new dependencies. Uses a `collections.deque` per key in memory. Fine for a single-process deployment. **`rate_limit.py`** (new file) ```python import time from collections import defaultdict, deque from fastapi import HTTPException, Request # shortcut: resets on restart, not shared across processes; switch to Redis when you scale out _calls: dict[str, deque] = defaultdict(deque) def rate_limit(max_calls: int = 10, window_seconds: int = 60): """Dependency: raises 429 when the caller exceeds max_calls in the window.""" def check(request: Request): key = request.client.host now = time.monotonic() window_start = now - window_seconds q = _calls[key] while q and q[0] < window_start: # drop timestamps outside the window q.popleft() if len(q) >= max_calls: raise HTTPException(status_code=429, detail="Too many requests") q.append(now) return check ``` **Your endpoint** — add one `Depends`: ```python from fastapi import Depends from rate_limit import rate_limit @app.get("/your-endpoint", dependencies=[Depends(rate_limit(max_calls=10, window_seconds=60))]) def your_endpoint(): ... ``` --- ### When to upgrade | Situation | Do this | |-----------|---------| | Multiple workers / containers | Replace `_calls` dict with Redis + a sorted set or `redis-py`'s `ZADD`/`ZREMRANGEBYSCORE` | | Auth'd users | Change `key` from `request.client.host` to the user ID from your token | | Different limits per route | Pass different `max_calls`/`window_seconds` to each `Depends(rate_limit(...))` | --- **Skipped:** a `Retry-After` header on the 429 (add `headers={"Retry-After": str(window_seconds)}` to the `HTTPException` if you want it). **Risk:** all processes on the same host share an IP behind a NAT — you may block innocent users if you set the limit too tight.

Why it failed: All assertions passed

History on main

email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 Now passing

Input

{"task":"Write me a Python function that validates email addresses."}

✗ main0.00

Use Python's built-in `email.utils.parseaddr`: ```python from email.utils import parseaddr def is_valid_email(email): name, addr = parseaddr(email) return '@' in addr and '.' in addr.split('@')[1] ``` That's it. It parses the email, checks for `@` and a dot after it (minimal RFC 5321). For 95% of use cases, this beats a regex or a dependency. **Skipped:** RFC 5322 full compliance, length limits, DNS MX checks, plus-addressing edge cases. **Risk:** accepts some invalid addresses (e.g., `user@localhost`); if you need strict validation, ask your backend to send a confirmation link instead—that's the only real check.

✓ This PR1.00

```python import re def is_valid_email(email): return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (overkill for 99% of uses), DNS MX record checks, confusing edge cases like quoted strings. This catches the 99% case: something@domain.extension. **Risk:** Doesn't reject some invalid emails like `a@b.c` (valid by this regex but rare in practice). If you actually need to *send* to it, the bounce tells you. If you need strict validation, use `email_validator` package instead—three lines become one import.

Why it failed: All assertions passed

History on main