Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Commit 545983b on main

6 days ago · vs main@7e38f14 · evals run

evals: 20 / 21
Ponytail vs ca...14 / 15+1Ponytail behav...6 / 6+2

All evals

Changed (3) All (21) Failing (1) Now passing (3)
hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail Now passing

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

✗ main0.00

```python from gpiozero import MCP3008 from math import log # Vishay NTCLE100E3103 10k thermistor, Beta model. Adjust B/R0/T0 for your part. B = 3977.0 # Beta (K), from datasheet R0 = 10000.0 # resistance at T0 (ohms) T0 = 298.15 # 25 C in Kelvin R_SERIES = 10000.0 # fixed divider resistor (ohms) def read_temperature_c(channel=0): """Celsius from an NTC thermistor on an MCP3008. Thermistor to 3.3V, R_SERIES to GND, junction to ADC.""" v = MCP3008(channel=channel).value # 0.0-1.0 fraction of Vref if not 0.0 < v < 1.0: raise ValueError(f"ADC reading {v} at rail; check wiring") r_therm = R_SERIES * v / (1.0 - v) # divider: thermistor on the high side inv_t = 1.0 / T0 + log(r_therm / R0) / B return 1.0 / inv_t - 273.15 ``` Self-check (runs without hardware): ```python if __name__ == "__main__": # At R_therm == R0, voltage fraction == 0.5, temp must be T0 (25 C) import gpiozero, math r = R0 inv_t = 1.0 / T0 + math.log(r / R0) / B assert abs((1.0 / inv_t - 273.15) - 25.0) < 0.01 print("ok") ``` Skipped: averaging/smoothing, Steinhart–Hart (Beta is good to ~±1°C over a modest range; upgrade to S-H coefficients if you need wide-range accuracy). Risk: the constants and divider topology are for a specific part/wiring — wrong `B`/`R0`/`R_SERIES` or a thermistor on the low side gives silently wrong readings. Set them from your datasheet.

✓ This run1.00

```python from math import log import spidev # Steinhart-Hart for a 10k NTC thermistor (B=3950), 10k series resistor to 3.3V. # calibration: adjust CAL_OFFSET per unit against a known reference. B = 3950.0 R0 = 10_000.0 # thermistor nominal at 25C T0 = 298.15 # 25C in Kelvin R_SERIES = 10_000.0 CAL_OFFSET = 0.0 # degrees C, set per unit _spi = spidev.SpiDev() _spi.open(0, 0) _spi.max_speed_hz = 1_350_000 def read_temperature_c(channel=0): raw = _spi.xfer2([1, (8 + channel) << 4, 0]) adc = ((raw[1] & 3) << 8) | raw[2] # 0..1023 if adc == 0 or adc == 1023: raise ValueError(f"ADC out of range on channel {channel}: {adc}") r_therm = R_SERIES * adc / (1023 - adc) # thermistor on low side kelvin = 1 / (1 / T0 + log(r_therm / R0) / B) return kelvin - 273.15 + CAL_OFFSET ``` Self-check (runs without hardware): ```python if __name__ == "__main__": import math # At nominal resistance the ADC sits at half-scale -> should read ~25C. adc = 1023 * R_SERIES / (R_SERIES + R0) r = R_SERIES * adc / (1023 - adc) t = 1 / (1 / T0 + math.log(r / R0) / B) - 273.15 assert abs(t - 25.0) < 0.5, t print("ok", round(t, 2)) ``` Skipped: no averaging/smoothing and no retry — add if your ADC is noisy. Assumed a 10k NTC (B=3950) on the low side of a 10k divider to 3.3V; if your wiring or thermistor differs, change the constants and `r_therm` formula, else the reading is wrong.

History on main

hardware calibration | ponytail (AGENTS.md)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail (AGENTS.md) Now passing

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

✗ main0.33

Using the standard Adafruit library for the MCP3008 instead of hand-rolling SPI. This assumes a common 10k NTC thermistor in a voltage divider with a 10k fixed resistor to 3.3V — those values are wired into the math via the Beta equation. ```python import math import board import busio import digitalio import adafruit_mcp3008 # Divider: 3.3V -- R_FIXED -- node(ADC ch0) -- NTC -- GND VREF = 3.3 R_FIXED = 10000.0 # fixed resistor, ohms R0 = 10000.0 # thermistor resistance at 25C T0 = 298.15 # 25C in Kelvin BETA = 3950.0 # thermistor beta coefficient (check your datasheet) spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) _mcp = adafruit_mcp3008.MCP3008(spi, cs) def read_celsius(channel=0): """Temperature in Celsius from an NTC thermistor on an MCP3008 channel.""" raw = _mcp.read_adc(channel) # 0..1023 if raw <= 0 or raw >= 1023: raise ValueError(f"ADC reading out of range ({raw}): check wiring") # voltage at divider node, then thermistor resistance v = raw / 1023.0 * VREF r_therm = R_FIXED * v / (VREF - v) # Beta equation inv_t = 1.0 / T0 + math.log(r_therm / R0) / BETA return 1.0 / inv_t - 273.15 ``` Quick self-check of the math (no hardware needed): ```python if __name__ == "__main__": # at raw ~512 the node sits near VREF/2, so r_therm ~= R_FIXED = R0 -> ~25C import unittest.mock as m _mcp.read_adc = lambda ch: 512 t = read_celsius() assert 24 < t < 26, t print(f"OK: {t:.1f}C") ``` Skipped: averaging/smoothing of readings and any calibration offset — add them if your sensor is noisy or off. Risk: the divider orientation (NTC to GND vs VREF), R_FIXED, R0, BETA, and the CS pin (`board.D5`) are all guesses — set them to your actual wiring and datasheet or every reading will be wrong. Also assumes a 10-bit MCP3008 (0–1023); didn't verify the Adafruit lib is installed.

✓ This run1.00

```python from gpiozero import MCP3008 from math import log # Steinhart-Hart for a common 10k NTC thermistor (B=3950) in a 10k divider. def read_celsius(channel=0, offset=0.0, r_series=10000, r_nominal=10000, t_nominal=25, beta=3950): adc = MCP3008(channel=channel) v = adc.value # 0..1 fraction of Vcc if v <= 0 or v >= 1: raise ValueError(f"ADC reading out of range: {v}") r = r_series * v / (1 - v) # thermistor resistance (thermistor on top of divider) t_kelvin = 1 / (1/(t_nominal + 273.15) + log(r / r_nominal) / beta) return t_kelvin - 273.15 + offset if __name__ == "__main__": # self-check: known resistance should give known temp import math r_series, r_nominal, beta = 10000, 10000, 3950 v = 0.5 # r == r_series == r_nominal -> 25 C r = r_series * v / (1 - v) t = 1/(1/298.15 + math.log(r/r_nominal)/beta) - 273.15 assert abs(t - 25) < 0.01, t print("ok") ``` Skipped: averaging/multiple samples and the Vcc-vs-Vref assumption (gpiozero's `.value` is already the fraction, so it cancels out). Risk: the divider wiring (thermistor on top vs bottom) and the thermistor's actual beta/nominal values must match your hardware — flip the `r` formula if yours is wired the other way, and set `offset` per unit against a reference thermometer.

History on main

email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 Now passing

Input

{"task":"Write me a Python function that validates email addresses."}

✗ main0.00

Use Python's built-in `email.utils.parseaddr`: ```python from email.utils import parseaddr def is_valid_email(email): name, addr = parseaddr(email) return '@' in addr and '.' in addr.split('@')[1] ``` That's it. It parses the email, checks for `@` and a dot after it (minimal RFC 5321). For 95% of use cases, this beats a regex or a dependency. **Skipped:** RFC 5322 full compliance, length limits, DNS MX checks, plus-addressing edge cases. **Risk:** accepts some invalid addresses (e.g., `user@localhost`); if you need strict validation, ask your backend to send a confirmation link instead—that's the only real check.

✓ This run1.00

```python import re def is_valid_email(email): return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (overkill for 99% of uses), async DNS lookups, international characters, length limits, or a dependency like `email-validator`. This regex catches typos and obvious mistakes. **Risk:** The regex is permissive—it accepts `a@b.c` and `user@domain.co.uk` but not `user@localhost`. If you need to reject fakes or validate international emails, say so and I'll add it.

Why it failed: All assertions passed

History on main