Run the promptfoo evals in CI with evalship #1
gglucass · evalship/setup → main · 6832ef3 (latest is b182120) ·
6 days ago
· evals run
46 of 51 evals pass, 1 couldn't run. This is the first run, so there's nothing to compare with: the 4 that fail are where your evals start, not regressions. Every later pull request is compared with main's results.
Notes
-
Couldn't run: 1 eval in
Ponytail vs caveman vs no-skillfailed before it was scored (ModuleNotFoundError: No module named 'email_validator').
All evals
test 086b3059Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / test 086b3059 | baseline (no skill) Unchanged
Input
{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}
Output
# Reading Temperature from a Thermistor via MCP3008 Here's a Python function to read temperature from a thermistor connected to an MCP3008 ADC on a Raspberry Pi. ## Prerequisites ```bash pip install adafruit-circuitpython-mcp3xxx ``` Enable SPI on your Pi via `sudo raspi-config` → Interface Options → SPI. ## Circuit Assumptions A common voltage divider setup: ``` 3.3V ---[ Series Resistor R ]---+--- MCP3008 CH0 | [ Thermistor ] | GND ``` ## Code ```python import math import board import busio import digitalio import adafruit_mcp3xxx.mcp3008 as MCP from adafruit_mcp3xxx.analog_in import AnalogIn def setup_mcp3008(channel=0): """Initialize MCP3008 and return the requested analog input channel.""" spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) # Chip-select pin (adjust as needed) mcp = MCP.MCP3008(spi, cs) channels = [MCP.P0, MCP.P1, MCP.P2, MCP.P3, MCP.P4, MCP.P5, MCP.P6, MCP.P7] return AnalogIn(mcp, channels[channel]) def read_temperature_celsius( adc_channel, v_supply=3.3, # Supply voltage (V) series_resistor=10000.0, # Series resistor in the divider (ohms) nominal_resistance=10000.0, # Thermistor resistance at nominal temp (ohms) nominal_temp=25.0, # Nominal temperature (°C) beta=3950.0, # Beta coefficient of the thermistor thermistor_on_ground=True, # True if thermistor is between ADC node and GND ): """ Read temperature in Celsius from a thermistor using the Beta (Steinhart simplified) equation. Parameters ---------- adc_channel : AnalogIn The MCP3008 analog input channel object. v_supply : float The voltage feeding the divider. series_resistor : float The fixed resistor value in ohms. nominal_resistance : float Thermistor resistance at the nominal temperature. nominal_temp : float Nominal temperature in Celsius (usually 25°C). beta : float The thermistor's Beta coefficient (from its datasheet). thermistor_on_ground : bool Orientation of the thermistor in the divider. Returns ------- float Temperature in degrees Celsius. """ # MCP3008 is 10-bit (0-1023); voltage read via AnalogIn voltage = adc_channel.voltage # already scaled to volts by the library # Avoid division by zero / invalid readings if voltage <= 0 or voltage >= v_supply: raise ValueError(f"Invalid ADC voltage reading: {voltage:.3f} V") # Calculate thermistor resistance from the voltage divider if thermistor_on_ground: # V_adc = V_supply * R_thermistor / (R_series + R_thermistor) r_thermistor = series_resistor * voltage / (v_supply - voltage) else: # Thermistor is on the high side r_thermistor = series_resistor * (v_supply - voltage) / voltage # Beta equation (simplified Steinhart-Hart) # 1/T = 1/T0 + (1/Beta) * ln(R/R0) (T in Kelvin) t0_kelvin = nominal_temp + 273.15 inv_t = (1.0 / t0_kelvin) + (1.0 / beta) * math.log(r_thermistor / nominal_resistance) temp_kelvin = 1.0 / inv_t return temp_kelvin - 273.15 # --------------------------------------------------------------------------- # Example usage # --------------------------------------------------------------------------- if __name__ == "__main__": import time ch0 = setup_mcp3008(channel=0) try: while True: temp_c = read_temperature_celsius( ch0, v_supply=3.3, series_resistor=10000.0, nominal_resistance=10000.0, nominal_temp=25.0, beta=3950.0, ) print(f"Temperature: {temp_c:.2f} °C") time.sleep(1.0) except KeyboardInterrupt: print("\nStopped.") ``` ## Key Notes 1. **Thermistor parameters** — Check your ther…
Why it failed: Treats the hardware as ideal; no calibration knob.
test 086b3059Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / test 086b3059 | ponytail Unchanged
Input
{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}
Output
```python from gpiozero import MCP3008 from math import log # Vishay NTCLE100E3103 / generic 10k NTC, Beta model R_FIXED = 10_000 # series resistor (ohms) R0 = 10_000 # thermistor resistance at 25 °C BETA = 3950 # thermistor beta (check your datasheet) T0 = 298.15 # 25 °C in kelvin def read_celsius(channel=0): adc = MCP3008(channel=channel) v = adc.value # 0..1 fraction of Vref if not 0 < v < 1: raise ValueError(f"ADC reading {v} out of range (open/short circuit?)") r = R_FIXED * v / (1 - v) # thermistor resistance, divider: 3V3 - R_FIXED - node - thermistor - GND kelvin = 1 / (1 / T0 + log(r / R0) / BETA) return kelvin - 273.15 ``` ```python # self-check: at R == R0 the reading must be 25 °C def _check(): r, kelvin = R0, 1 / (1 / T0 + log(R0 / R0) / BETA) assert abs(kelvin - 273.15 - 25.0) < 0.01 _check() ``` Skipped: averaging/smoothing and config flags — add if your readings are noisy. **You must set `BETA`, `R0`, `R_FIXED` and the divider wiring to match your hardware; wrong values give wrong temps silently.** Steinhart–Hart (3 coefficients) beats Beta if you need accuracy over a wide range. Didn't run it (no Pi/gpiozero here).
Why it failed: Treats the hardware as ideal; no calibration knob.
test 388a1256Ponytail behavior gates: hardware calibration, requested explanation, one runnable check. / test 388a1256 | baseline (no skill) Unchanged
Input
{"probe":"onecheck","task":"Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."}
Output
# Duration String Parser Here's a Python function that parses duration strings like `'1h30m45s'` into total seconds: ```python import re def parse_duration(duration: str) -> int: """ Parse a duration string into total seconds. Supports units: d (days), h (hours), m (minutes), s (seconds). Example: '1h30m45s' -> 5445 Args: duration: A duration string like '1h30m45s', '2d', '90m'. Returns: Total number of seconds as an integer. Raises: ValueError: If the string is empty or contains invalid format. """ if not duration or not duration.strip(): raise ValueError("Duration string cannot be empty") duration = duration.strip().lower() # Multipliers for each unit (in seconds) multipliers = { 'd': 86400, 'h': 3600, 'm': 60, 's': 1, } # Match groups of digits followed by a unit character pattern = re.compile(r'(\d+)([dhms])') total = 0 matched_length = 0 for match in pattern.finditer(duration): value, unit = match.groups() total += int(value) * multipliers[unit] matched_length += len(match.group(0)) # Ensure the entire string was consumed (no invalid characters) if matched_length != len(duration): raise ValueError(f"Invalid duration format: {duration!r}") return total ``` ## Usage Examples ```python print(parse_duration('1h30m45s')) # 5445 print(parse_duration('2d')) # 172800 print(parse_duration('90m')) # 5400 print(parse_duration('45s')) # 45 print(parse_duration('1d2h3m4s')) # 93784 # Errors try: parse_duration('abc') except ValueError as e: print(e) # Invalid duration format: 'abc' ``` ## How It Works 1. **Validation**: Rejects empty/whitespace-only strings. 2. **Regex matching**: `(\d+)([dhms])` captures a number paired with a unit. 3. **Accumulation**: Each `value × multiplier` is summed into the total. 4. **Integrity check**: Comparing `matched_length` to the full string length ensures there are no stray or invalid characters (e.g., `'1h30x'` fails). ## Optional: Support for Floats If you want to allow fractional values (e.g., `'1.5h'`): ```python def parse_duration_float(duration: str) -> float: if not duration or not duration.strip(): raise ValueError("Duration string cannot be empty") duration = duration.strip().lower() multipliers = {'d': 86400, 'h': 3600, 'm': 60, 's': 1} pattern = re.compile(r'(\d+(?:\.\d+)?)([dhms])') total = 0.0 matched_length = 0 for match in pattern.finditer(duration): value, unit = match.groups() total += float(value) * multipliers[unit] matched_length += len(match.group(0)) if matched_length != len(duration): raise ValueError(f"Invalid duration format: {duration!r}") return total print(parse_duration_float('1.5h')) # 5400.0 ``` Let me know if you need support for additional units (weeks, milliseconds) or different formats!
Why it failed: No runnable check left behind.
test d54feca0Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / test d54feca0 | baseline (no skill) | openrouter:anthropic/claude-sonnet-4.6 Unchanged
Input
{"task":"Write me a Python function that validates email addresses."}
Output
## Email Validation Function Here's a robust email validation function with multiple levels of checking: ```python import re def validate_email(email: str) -> dict[str, bool | str]: """ Validates an email address with multiple checks. Args: email: The email address to validate. Returns: A dictionary with: - 'is_valid': Boolean indicating if the email is valid. - 'message': A descriptive message about the validation result. """ # Check if email is a non-empty string if not isinstance(email, str) or not email.strip(): return {"is_valid": False, "message": "Email must be a non-empty string."} email = email.strip() # Check overall length (max 254 characters per RFC 5321) if len(email) > 254: return {"is_valid": False, "message": "Email exceeds maximum length of 254 characters."} # Check for exactly one '@' symbol if email.count("@") != 1: return {"is_valid": False, "message": "Email must contain exactly one '@' symbol."} local, domain = email.split("@") # --- Local part checks --- if not local: return {"is_valid": False, "message": "Local part (before '@') cannot be empty."} if len(local) > 64: return {"is_valid": False, "message": "Local part exceeds maximum length of 64 characters."} if local.startswith(".") or local.endswith("."): return {"is_valid": False, "message": "Local part cannot start or end with a dot."} if ".." in local: return {"is_valid": False, "message": "Local part cannot contain consecutive dots."} local_pattern = r'^[a-zA-Z0-9._%+\-]+$' if not re.match(local_pattern, local): return {"is_valid": False, "message": "Local part contains invalid characters."} # --- Domain part checks --- if not domain: return {"is_valid": False, "message": "Domain part (after '@') cannot be empty."} if domain.startswith("-") or domain.endswith("-"): return {"is_valid": False, "message": "Domain cannot start or end with a hyphen."} if ".." in domain: return {"is_valid": False, "message": "Domain cannot contain consecutive dots."} if "." not in domain: return {"is_valid": False, "message": "Domain must contain at least one dot."} domain_pattern = r'^[a-zA-Z0-9.\-]+$' if not re.match(domain_pattern, domain): return {"is_valid": False, "message": "Domain contains invalid characters."} # Check TLD (last part after the final dot) tld = domain.split(".")[-1] if len(tld) < 2: return {"is_valid": False, "message": "Top-level domain must be at least 2 characters."} if not tld.isalpha(): return {"is_valid": False, "message": "Top-level domain must contain only letters."} return {"is_valid": True, "message": "Email is valid."} def is_valid_email(email: str) -> bool: """Simplified wrapper that returns only a boolean.""" return validate_email(email)["is_valid"] ``` --- ### Example Usage ```python test_emails = [ "user@example.com", # Valid "user.name+tag@domain.co", # Valid "invalid-email", # No @ symbol "@nodomain.com", # Empty local part "user@", # Empty domain "user@@domain.com", # Multiple @ symbols "user@domain", # No TLD "user@domain.c", # TLD too short ".user@domain.com", # Local starts with dot "user..name@domain.com", # Consecutive dots "a" * 65 + "@domain.com", # Local part too long ] for email in test_emails: result = validate_email(email) status = "✅" if result["is_valid"] else "❌" print(f"{status} {email!r}: {result['message']}") ``` **Output:** ``` ✅ 'user@example.com': Email is valid. ✅ 'user.name+tag@domain.co': Email is valid. ❌ 'invalid-email': Email must contain exactly one '@' symbol. ❌ '@nodomain.com': Local part (before '@') cannot be empty. ❌ 'user@': Domain part (after '@') cannot be empty. ❌ 'user@@domain.com': Email m…
Why it failed: FAIL: accepted invalid: no-at-sign; accepted invalid: empty string; accepted invalid: @missing-local.com
test d54feca0Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / test d54feca0 | ponytail | openrouter:anthropic/claude-haiku-4.5 Unchanged
Input
{"task":"Write me a Python function that validates email addresses."}
Output
Don't write one. Use this: ```python from email_validator import validate_email def validate_email_address(email): try: validate_email(email) return True except: return False ``` Install it: `pip install email_validator` **Why:** The RFC for email is 300+ pages. Hand-rolled regex fails on real addresses (subdomains, plus addressing, quoted strings). This library handles the hard cases and is already in most projects. **Skipped:** custom regex, edge case handling, DNS verification. **Risk:** if you need to reject addresses your mail provider won't accept, test against your actual provider first—this validator is strict-correct, not provider-matched.
Why it failed: Traceback (most recent call last): File "/tmp/ponytail-bench-1793271147914-d7xco5b18gf.py", line 2, in <module> from email_validator import validate_email ModuleNotFoundError: No module named 'email_validator'