Run the promptfoo evals in CI with evalship #1
gglucass · evalship/setup → main · b182120 ·
6 days ago
· evals run
46 of 53 evals pass, 2 couldn't run. main has no eval results yet, so there's nothing to compare with: the 5 that fail are where your evals start, not regressions. Every later pull request is compared with main's results. No eval flaws found in this pull request.
Notes
-
Couldn't run: 2 evals in
Ponytail behavior gatesandPonytail vs caveman vs no-skillfailed before they were scored (API error: 403 Forbidden).
0 eval flaws in this pull request
Eval note in .github/workflows/evals.yml, dismissed by gglucass as won't fix: setup repeats only the cases evalship lists as flaky, once main's runs show them flipping, not every single-shot case (.github/workflows/evals.yml:31)
Single-shot runs each case once at temperature 1 (promptfooconfig.yaml), while behavior gets --repeat 3 for that same sampling noise, so one sampled miss on the correctness gate can register as a new failure. Confirm evalship's flake handling covers single runs, or add --repeat to this job's flags.
All evals
hardware calibrationPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration Couldn't run
Input
{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}
Outputs weren't recorded for this eval. Record them to see what the model said.
Why it failed: API error: 403 Forbidden {"error":{"message":"Key limit exceeded (weekly limit). Manage it using https://openrouter.ai/workspaces/default/keys/e9d4012025061a095e389f738ed10eb135eb2a8c31ac5df3a03c2390348d9a86 $3.22 of $3.00 used; resets 2026-11-02T00:00:00.000Z.","code":403,"metadata":{"limit_source":"openrouter_key_limit","remedy_hint":"Raise or remove this API key's usage limit at https://openrouter.ai/settings/keys, or use a key without a limit.","scope":"key_limit","limit_usd":3,"spend_usd":3.223687,"resets_at":"2026-11-02T00:00:00.000Z"}}}
email validatorPonytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator Couldn't run
Input
{"task":"Write me a Python function that validates email addresses."}
Outputs weren't recorded for this eval. Record them to see what the model said.
Why it failed: API error: 403 Forbidden {"error":{"message":"Key limit exceeded (weekly limit). Manage it using https://openrouter.ai/workspaces/default/keys/e9d4012025061a095e389f738ed10eb135eb2a8c31ac5df3a03c2390348d9a86 $3.22 of $3.00 used; resets 2026-11-02T00:00:00.000Z.","code":403,"metadata":{"limit_source":"openrouter_key_limit","remedy_hint":"Raise or remove this API key's usage limit at https://openrouter.ai/settings/keys, or use a key without a limit.","scope":"key_limit","limit_usd":3,"spend_usd":3.223687,"resets_at":"2026-11-02T00:00:00.000Z"}}}
hardware calibration | baseline (no skill)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | baseline (no skill) Unchanged
Input
{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}
Output
# Reading Temperature from a Thermistor via MCP3008 Here's a complete Python function to read temperature from a thermistor connected to an MCP3008 ADC on a Raspberry Pi. ## Prerequisites ```bash pip install adafruit-circuitpython-mcp3xxx ``` Enable SPI on your Raspberry Pi via `sudo raspi-config` → Interface Options → SPI. ## Code ```python import math import time import board import busio import digitalio import adafruit_mcp3xxx.mcp3008 as MCP from adafruit_mcp3xxx.analog_in import AnalogIn def read_thermistor_temp( channel=MCP.P0, vcc=3.3, series_resistor=10000.0, # Fixed resistor in the voltage divider (ohms) nominal_resistance=10000.0, # Thermistor resistance at nominal temp (ohms) nominal_temp_c=25.0, # Nominal temperature (Celsius) beta_coefficient=3950.0, # Beta value from thermistor datasheet ): """ Read temperature in Celsius from a thermistor via MCP3008 channel 0. Wiring (voltage divider): VCC --- [series_resistor] ---+--- [thermistor] --- GND | MCP3008 CH0 Returns: Temperature in degrees Celsius (float). """ # --- Set up SPI bus and MCP3008 --- spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) # Chip-select pin (adjust as needed) mcp = MCP.MCP3008(spi, cs) # Create an analog input on the specified channel chan = AnalogIn(mcp, channel) # --- Read the ADC voltage --- # chan.voltage already gives the measured voltage at the ADC pin v_out = chan.voltage if v_out <= 0: raise ValueError("Measured voltage is zero; check wiring.") # --- Compute thermistor resistance --- # Voltage divider: v_out = vcc * (R_therm / (R_series + R_therm)) # Solve for R_therm: r_therm = series_resistor * (v_out / (vcc - v_out)) # --- Apply the Beta (Steinhart) equation --- # 1/T = 1/T0 + (1/B) * ln(R/R0) # Temperatures in Kelvin t0_kelvin = nominal_temp_c + 273.15 inv_t = (1.0 / t0_kelvin) + (1.0 / beta_coefficient) * math.log( r_therm / nominal_resistance ) temp_kelvin = 1.0 / inv_t temp_celsius = temp_kelvin - 273.15 return temp_celsius if __name__ == "__main__": while True: try: temp = read_thermistor_temp() print(f"Temperature: {temp:.2f} °C") except Exception as e: print(f"Error reading temperature: {e}") time.sleep(1) ``` ## Notes **1. Wiring matters for the math** The code assumes this voltage divider (thermistor on the low/GND side): ``` VCC --- [R_series] ---+--- CH0 | [Thermistor] --- GND ``` If you swap the thermistor and resistor positions, change the resistance calculation to: ```python r_therm = series_resistor * ((vcc - v_out) / v_out) ``` **2. Tune the parameters to your hardware** Grab these from your thermistor's datasheet: - `nominal_resistance` — e.g. 10 kΩ (common NTC) - `nominal_temp_c` — usually 25 °C - `beta_coefficient` — e.g. 3950 (varies by part) **3. Efficiency improvement** For repeated reads, initialize the SPI/MCP objects **once** instead of per-call: ```python class ThermistorReader: def __init__(self, channel=MCP.P0, cs_pin=board.D5, **params): spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(cs_pin) self.mcp = MCP.MCP3008(spi, cs) self.chan = AnalogIn(self.mcp, channel) self.params = params def read_temp(self): v_out = self.chan.voltage r_therm = self.params.get("series_resistor", 10000.0) * ( v_out / (self.params.get("vcc", 3.3) - v_out) ) t0 = self.params.get("nominal_temp_c", 25.0) + 273.15 beta = self.params.get("beta_coefficient", 3950.0) r0 = self.params.get("nominal_resistance", 10000.0) inv_t = (1 / t0) + (1 / beta) * math.log(r_therm / r0) return (1 / inv_t) - 273.15 ``` **4…
Why it failed: Treats the hardware as ideal; no calibration knob.
hardware calibration | ponytailPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail Unchanged
Input
{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}
Output
```python import spidev import math # Beta-model thermistor on MCP3008 CH0. 3.3V ref, 10kΩ series resistor, 10kΩ NTC @ 25°C, B=3950. # Calibrate R_SERIES, R0, T0, B to your actual parts. VREF = 3.3 R_SERIES = 10000.0 R0 = 10000.0 T0 = 298.15 # 25°C in Kelvin BETA = 3950.0 _spi = spidev.SpiDev() _spi.open(0, 0) _spi.max_speed_hz = 1350000 def read_temperature_c(channel=0): raw = _spi.xfer2([1, (8 + channel) << 4, 0]) adc = ((raw[1] & 3) << 8) | raw[2] # 0..1023 if adc == 0 or adc == 1023: raise ValueError(f"ADC reading {adc} out of range (open/short thermistor?)") voltage = adc / 1023.0 * VREF r_therm = R_SERIES * voltage / (VREF - voltage) # thermistor on low side, series to VREF kelvin = 1.0 / (1.0 / T0 + math.log(r_therm / R0) / BETA) return kelvin - 273.15 ``` Wiring this assumes: 3.3V → R_SERIES → CH0 → thermistor → GND. If yours is flipped, swap the `r_therm` formula to `R_SERIES * (VREF - voltage) / voltage`. Skipped: averaging multiple samples (add later if readings jitter), and the Steinhart-Hart 3-coefficient model (Beta is fine for a ±5°C hobby range). Not checked: your actual B/R0 values — the defaults are placeholders you must calibrate, or temps will be wrong.
Why it failed: Treats the hardware as ideal; no calibration knob.
one runnable check | baseline (no skill)Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / one runnable check | baseline (no skill) Unchanged
Input
{"probe":"onecheck","task":"Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."}
Output
# Duration String Parser Here's a Python function that parses duration strings like `'1h30m45s'` into total seconds: ```python import re def parse_duration(duration_str): """ Parse a duration string into total seconds. Supports the following units: - 'd' (days) - 'h' (hours) - 'm' (minutes) - 's' (seconds) Examples: '1h30m45s' -> 5445 '2d' -> 172800 '90m' -> 5400 '45s' -> 45 Args: duration_str (str): The duration string to parse. Returns: int: Total number of seconds. Raises: ValueError: If the string is empty or contains invalid tokens. """ if not duration_str or not duration_str.strip(): raise ValueError("Duration string cannot be empty") duration_str = duration_str.strip().lower() # Mapping of unit characters to their value in seconds unit_seconds = { 'd': 86400, 'h': 3600, 'm': 60, 's': 1, } # Match number + unit pairs, e.g. "1h", "30m", "45s" pattern = re.compile(r'(\d+)([dhms])') matches = pattern.findall(duration_str) # Verify the entire string was consumed (no leftover/invalid characters) consumed = ''.join(num + unit for num, unit in matches) if consumed != duration_str: raise ValueError(f"Invalid duration string: {duration_str!r}") total_seconds = 0 for value, unit in matches: total_seconds += int(value) * unit_seconds[unit] return total_seconds ``` ## Usage Examples ```python print(parse_duration('1h30m45s')) # 5445 print(parse_duration('2d')) # 172800 print(parse_duration('90m')) # 5400 print(parse_duration('45s')) # 45 print(parse_duration('1d2h3m4s')) # 93784 # Invalid inputs raise ValueError try: parse_duration('1x30y') except ValueError as e: print(e) # Invalid duration string: '1x30y' ``` ## How It Works 1. **Normalization** — Strips whitespace and lowercases the input. 2. **Regex matching** — `(\d+)([dhms])` captures each number/unit pair. 3. **Validation** — Reconstructs the matched string and compares it to the original to ensure there are no invalid or leftover characters. 4. **Summation** — Multiplies each value by its corresponding factor in seconds. ## Optional: Support for Fractional Values If you want to allow decimal values like `'1.5h'`: ```python def parse_duration_float(duration_str): """Parse a duration string with optional decimal values into total seconds.""" if not duration_str or not duration_str.strip(): raise ValueError("Duration string cannot be empty") duration_str = duration_str.strip().lower() unit_seconds = {'d': 86400, 'h': 3600, 'm': 60, 's': 1} pattern = re.compile(r'(\d+(?:\.\d+)?)([dhms])') matches = pattern.findall(duration_str) consumed = ''.join(num + unit for num, unit in matches) if consumed != duration_str: raise ValueError(f"Invalid duration string: {duration_str!r}") return sum(float(value) * unit_seconds[unit] for value, unit in matches) print(parse_duration_float('1.5h')) # 5400.0 ``` Let me know if you'd like it to support additional units (like weeks or milliseconds) or return a `timedelta` object instead!
Why it failed: No runnable check left behind.
FastAPI rate-limit | ponytail | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / FastAPI rate-limit | ponytail | openrouter:anthropic/claude-sonnet-4.6 Unchanged
Input
{"task":"Add rate limiting to my FastAPI endpoint so users can't spam it."}
Output
**What's your setup?** Before I write anything, I need to know: 1. **How do you identify users?** (API key in header, JWT, session cookie, IP address?) 2. **What limit?** (e.g. 10 req/min per user) 3. **Do you have Redis?** (or is this a single-process app where in-memory is fine?) --- While you answer — the honest shortlist: | Option | When it fits | |---|---| | **`slowapi`** (1 decorator) | Already in your deps, or you want standard `Limits` header support | | **In-memory dict + middleware** | Single process, no Redis, no install | | **Reverse proxy rule** (nginx/Caddy/Cloudflare) | You control infra — zero app code | The reverse-proxy option is free and the most robust. If that's available, it beats adding code entirely. --- **Skipped:** the actual implementation — answering blind would likely give you the wrong key, wrong limit, or an approach that breaks under your auth model. **Risk:** in-memory rate limiting resets on restart and doesn't work across multiple workers/replicas.
Why it failed: Missing: no FastAPI usage
email validator | baseline (no skill) | openrouter:anthropic/claude-sonnet-4.6Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | baseline (no skill) | openrouter:anthropic/claude-sonnet-4.6 Unchanged
Input
{"task":"Write me a Python function that validates email addresses."}
Output
## Email Validation Function Here's a Python function that validates email addresses using both regex and logical checks: ```python import re def validate_email(email: str) -> dict: """ Validates an email address and returns a detailed result. Args: email: The email address to validate Returns: A dict with 'is_valid' (bool) and 'message' (str) keys """ # Check if email is a non-empty string if not isinstance(email, str) or not email: return {"is_valid": False, "message": "Email must be a non-empty string"} # Check length (max 254 characters per RFC 5321) if len(email) > 254: return {"is_valid": False, "message": "Email exceeds maximum length of 254 characters"} # Check for exactly one @ symbol if email.count("@") != 1: return {"is_valid": False, "message": "Email must contain exactly one '@' symbol"} local, domain = email.split("@") # Validate local part (before @) if not local: return {"is_valid": False, "message": "Local part (before '@') cannot be empty"} if len(local) > 64: return {"is_valid": False, "message": "Local part exceeds maximum length of 64 characters"} # Validate domain part (after @) if not domain: return {"is_valid": False, "message": "Domain part (after '@') cannot be empty"} if "." not in domain: return {"is_valid": False, "message": "Domain must contain at least one '.'"} if domain.startswith(".") or domain.endswith("."): return {"is_valid": False, "message": "Domain cannot start or end with a '.'"} # Full regex pattern for thorough validation pattern = r"^[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}$" if not re.match(pattern, email): return {"is_valid": False, "message": "Email format is invalid"} return {"is_valid": True, "message": "Email is valid"} def is_valid_email(email: str) -> bool: """Simple wrapper that returns just a boolean.""" return validate_email(email)["is_valid"] ``` ### Usage Examples ```python test_emails = [ "user@example.com", # Valid "user.name+tag@domain.co", # Valid "invalid-email", # No @ symbol "@domain.com", # Empty local part "user@", # Empty domain "user@domain", # No TLD "user@@domain.com", # Multiple @ symbols "a" * 65 + "@domain.com", # Local part too long "user@.domain.com", # Domain starts with dot ] for email in test_emails: result = validate_email(email) status = "✅" if result["is_valid"] else "❌" print(f"{status} '{email}': {result['message']}") ``` ### Output ``` ✅ 'user@example.com': Email is valid ✅ 'user.name+tag@domain.co': Email is valid ❌ 'invalid-email': Email must contain exactly one '@' symbol ❌ '@domain.com': Local part (before '@') cannot be empty ❌ 'user@': Domain part (after '@') cannot be empty ❌ 'user@domain': Domain must contain at least one '.' ❌ 'user@@domain.com': Email must contain exactly one '@' symbol ❌ 'aaaaa...@domain.com': Local part exceeds maximum length of 64 characters ❌ 'user@.domain.com': Domain cannot start or end with a '.' ``` ### Key Features | Feature | Details | |---|---| | **Type check** | Ensures input is a non-empty string | | **Length limits** | 254 chars total, 64 chars for local part (RFC 5321) | | **Structure checks** | Exactly one `@`, valid domain format | | **Regex validation** | Catches remaining format issues | | **Detailed feedback** | Returns specific error messages | > **Note:** For production use, consider a library like **`email-validator`** (`pip install email-validator`) which handles edge cases and checks DNS records for domain existence.
Why it failed: FAIL: accepted invalid: no-at-sign; accepted invalid: empty string; accepted invalid: @missing-local.com