Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Run the promptfoo evals in CI with evalship #1

gglucass · evalship/setup → main · a83e85d (latest is b182120) · 6 days ago · evals run

evals: 46 / 51eval flaws: 1
Ponytail behav...3 / 6Ponytail vs ca...43 / 45

46 of 51 evals pass, 1 couldn't run. main has no eval results yet, so there's nothing to compare with: the 4 that fail are where your evals start, not regressions. Every later pull request is compared with main's results. 1 eval flaw found in this pull request.

Notes

  • Couldn't run: 1 eval in Ponytail vs caveman vs no-skill failed before it was scored (ModuleNotFoundError: No module named 'email_validator').

1 eval flaw in this pull request

If promptfoo exits before writing ../eval-results.json, the eval step's continue-on-error hides the failure and no step checks that the file exists, so evalship gets no results for that run. Add a step after the eval that fails when eval-results.json is missing.

Fix instructions for your coding agent
evalship's review of pull request #1 found this flaw in our LLM evals: If promptfoo exits before writing ../eval-results.json, the eval step's continue-on-error hides the failure and no step checks that the file exists, so evalship gets no results for that run. Add a step after the eval that fails when eval-results.json is missing. See .github/workflows/evals.yml:62.
Read the code it names first. If it's a mistake, fix it on a new branch from main. If the change is deliberate, don't undo it: say what the evals can no longer catch, and suggest a case or check that would.
Eval note in .github/workflows/evals.yml, dismissed by gglucass as won't fix: setup repeats only the cases evalship lists as flaky, once main's runs show them flipping, not every single-shot case (.github/workflows/evals.yml:31)

The behavior and single-shot suites now feed evalship's check on pull requests from this repo, but open findings "Two behavior gates pass replies that skip the behavior" and "The rate-limit check passes an endpoint with no rate limiting" mean some of those passes don't show the behavior or the rate limit they test. Fix both gates before that check is required. See your eval review.

All evals

All (51) Failing (5)
hardware calibrationPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | baseline (no skill) Unchanged

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

# Reading Temperature from a Thermistor via MCP3008 Here's a complete Python function to read temperature from a thermistor connected to an MCP3008 ADC on a Raspberry Pi. ## Prerequisites ```bash pip install adafruit-circuitpython-mcp3xxx ``` Enable SPI on your Pi via `sudo raspi-config` → Interface Options → SPI. ## Circuit Assumptions A typical voltage-divider setup: ``` 3.3V ──[ Series Resistor R_fixed ]──┬── MCP3008 CH0 │ [ Thermistor ] │ GND ``` ## The Code ```python import math import board import busio import digitalio import adafruit_mcp3xxx.mcp3008 as MCP from adafruit_mcp3xxx.analog_in import AnalogIn def read_thermistor_celsius( channel=MCP.P0, v_supply=3.3, r_fixed=10000.0, # series resistor in ohms r_nominal=10000.0, # thermistor resistance at nominal temp t_nominal=25.0, # nominal temperature in Celsius beta=3950.0, # Beta coefficient of the thermistor thermistor_on_top=False, # True if thermistor is between V+ and CH0 ): """ Read temperature in Celsius from a thermistor on MCP3008 channel 0. Uses the Beta-parameter (simplified Steinhart-Hart) equation. Returns: Temperature in degrees Celsius (float). """ # Set up SPI bus and MCP3008 spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(board.D5) # Chip-select pin (adjust as needed) mcp = MCP.MCP3008(spi, cs) chan = AnalogIn(mcp, channel) # Read raw ADC value (0–65535 from the CircuitPython library) raw = chan.value if raw <= 0 or raw >= 65535: raise ValueError(f"ADC reading out of range: {raw}") # Convert raw reading to a fraction of full scale ratio = raw / 65535.0 # Compute the thermistor resistance from the voltage divider if thermistor_on_top: # V+ --[thermistor]-- CH0 --[R_fixed]-- GND r_therm = r_fixed * (1.0 / ratio - 1.0) else: # V+ --[R_fixed]-- CH0 --[thermistor]-- GND r_therm = r_fixed * ratio / (1.0 - ratio) # Beta equation: 1/T = 1/T0 + (1/B) * ln(R/R0) t0_kelvin = t_nominal + 273.15 inv_t = (1.0 / t0_kelvin) + (1.0 / beta) * math.log(r_therm / r_nominal) temp_kelvin = 1.0 / inv_t temp_celsius = temp_kelvin - 273.15 return temp_celsius if __name__ == "__main__": import time while True: try: temp = read_thermistor_celsius() print(f"Temperature: {temp:.2f} °C") except ValueError as e: print(f"Read error: {e}") time.sleep(1) ``` ## A More Efficient Version Re-initializing SPI on every call is wasteful. Here's a class-based approach that sets up the hardware once: ```python import math import board import busio import digitalio import adafruit_mcp3xxx.mcp3008 as MCP from adafruit_mcp3xxx.analog_in import AnalogIn class Thermistor: def __init__( self, channel=MCP.P0, cs_pin=board.D5, r_fixed=10000.0, r_nominal=10000.0, t_nominal=25.0, beta=3950.0, thermistor_on_top=False, samples=5, ): spi = busio.SPI(clock=board.SCK, MISO=board.MISO, MOSI=board.MOSI) cs = digitalio.DigitalInOut(cs_pin) self._mcp = MCP.MCP3008(spi, cs) self._chan = AnalogIn(self._mcp, channel) self.r_fixed = r_fixed self.r_nominal = r_nominal self.t_nominal = t_nominal self.beta = beta self.thermistor_on_top = thermistor_on_top self.samples = samples def read_celsius(self): # Average several readings to reduce noise total = sum(self._chan.value for _ in range(self.samples)) raw = total / self.samples ratio = raw / 65535.0 if not (0 < ratio < 1): raise ValueError(f"ADC ratio out of range: {ratio}") if self.thermistor_on_top: r_therm = self.r_fixed * (1.0 …

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

hardware calibrationPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration | ponytail Unchanged

Input

{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}

Output

A thermistor doesn't give temperature directly — the ADC reads a voltage, which you convert via the Steinhart-Hart equation. So this needs the thermistor's specs. I'll use common defaults (10k NTC, Beta 3950, 10k series resistor) as named constants you edit for your part. ```python import math from gpiozero import MCP3008 # For a 10k NTC thermistor, Beta model. Edit these for your part. R_SERIES = 10_000.0 # series resistor (ohms) R0 = 10_000.0 # thermistor resistance at T0 T0 = 298.15 # 25 °C in kelvin BETA = 3950.0 def read_celsius(channel=0): adc = MCP3008(channel=channel) # value is 0.0–1.0 fraction of Vref ratio = adc.value if not 0.0 < ratio < 1.0: raise ValueError(f"ADC out of range ({ratio}); check wiring") # Divider: thermistor on top, R_SERIES to ground. r_therm = R_SERIES * ratio / (1.0 - ratio) # Beta equation t_kelvin = 1.0 / (1.0 / T0 + math.log(r_therm / R0) / BETA) return t_kelvin - 273.15 ``` Self-check (no hardware needed — uses the math directly): ```python if __name__ == "__main__": # At 25 °C the divider sits at ratio 0.5 (R_therm == R_SERIES). t = 1.0 / (1.0 / T0 + math.log(1.0) / BETA) - 273.15 assert abs(t - 25.0) < 0.01, t print("ok") ``` Skipped: averaging multiple reads (NTC is noisy — add a loop if your readings jitter), and the thermistor-on-bottom wiring (flip the divider formula). Risk: the constants above are placeholders — wrong Beta/R0/wiring position gives plausible-looking but wrong temperatures, so set them from your part's datasheet.

Why it failed: Treats the hardware as ideal; no calibration knob.

History on main

one runnable checkPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / one runnable check | baseline (no skill) Unchanged

Input

{"probe":"onecheck","task":"Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."}

Output

# Duration String Parser Here's a Python function that parses duration strings like `'1h30m45s'` into total seconds: ```python import re def parse_duration(duration: str) -> int: """ Parse a duration string into total seconds. Supports the following units: - 'd' for days - 'h' for hours - 'm' for minutes - 's' for seconds Examples: '1h30m45s' -> 5445 '2d' -> 172800 '90m' -> 5400 '45s' -> 45 Args: duration: A string like '1h30m45s'. Returns: Total number of seconds as an integer. Raises: ValueError: If the string is empty or contains invalid characters. """ if not duration or not duration.strip(): raise ValueError("Duration string cannot be empty") # Mapping of unit characters to their value in seconds unit_seconds = { 'd': 86400, 'h': 3600, 'm': 60, 's': 1, } # Match number + unit pairs (e.g. '1h', '30m') pattern = re.compile(r'(\d+)([dhms])') matches = pattern.findall(duration) # Verify the entire string was consumed (no invalid parts) consumed = ''.join(num + unit for num, unit in matches) if consumed != duration.strip(): raise ValueError(f"Invalid duration string: {duration!r}") total = 0 for value, unit in matches: total += int(value) * unit_seconds[unit] return total ``` ## Usage Examples ```python print(parse_duration('1h30m45s')) # 5445 print(parse_duration('2d')) # 172800 print(parse_duration('90m')) # 5400 print(parse_duration('45s')) # 45 print(parse_duration('1d2h3m4s')) # 93784 ``` ## How It Works 1. **Regex matching**: `(\d+)([dhms])` captures a number followed by a unit character. `findall` returns a list of `(number, unit)` tuples. 2. **Validation**: We reconstruct the string from matched parts and compare to the original to catch invalid input (e.g., `'1x'` or `'abc'`). 3. **Summation**: Each value is multiplied by its unit's second count and summed. ## Optional: Support for Floats If you want to allow fractional values like `'1.5h'`, modify the pattern and return type: ```python def parse_duration_float(duration: str) -> float: if not duration or not duration.strip(): raise ValueError("Duration string cannot be empty") unit_seconds = {'d': 86400, 'h': 3600, 'm': 60, 's': 1} pattern = re.compile(r'(\d+(?:\.\d+)?)([dhms])') matches = pattern.findall(duration) consumed = ''.join(num + unit for num, unit in matches) if consumed != duration.strip(): raise ValueError(f"Invalid duration string: {duration!r}") return sum(float(value) * unit_seconds[unit] for value, unit in matches) ``` Let me know if you'd like to add support for weeks, milliseconds, or negative durations!

Why it failed: No runnable check left behind.

History on main

email validatorPonytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | baseline (no skill) | openrouter:anthropic/claude-sonnet-4.6 Unchanged

Input

{"task":"Write me a Python function that validates email addresses."}

Output

## Email Validation Function Here's a Python function that validates email addresses using both regex pattern matching and logical checks: ```python import re def validate_email(email: str) -> dict: """ Validates an email address and returns a detailed result. Args: email: The email address to validate. Returns: A dict with 'is_valid' (bool) and 'message' (str) keys. """ # Basic type and empty check if not isinstance(email, str): return {"is_valid": False, "message": "Email must be a string."} if not email.strip(): return {"is_valid": False, "message": "Email cannot be empty."} email = email.strip() # Check overall length (max 254 characters per RFC 5321) if len(email) > 254: return {"is_valid": False, "message": "Email exceeds maximum length of 254 characters."} # Check for exactly one @ symbol if email.count("@") != 1: return {"is_valid": False, "message": "Email must contain exactly one '@' symbol."} local, domain = email.split("@") # --- Local part checks --- if not local: return {"is_valid": False, "message": "Local part (before '@') cannot be empty."} if len(local) > 64: return {"is_valid": False, "message": "Local part exceeds maximum length of 64 characters."} # --- Domain part checks --- if not domain: return {"is_valid": False, "message": "Domain part (after '@') cannot be empty."} if "." not in domain: return {"is_valid": False, "message": "Domain must contain at least one dot."} if domain.startswith(".") or domain.endswith("."): return {"is_valid": False, "message": "Domain cannot start or end with a dot."} if domain.startswith("-") or domain.endswith("-"): return {"is_valid": False, "message": "Domain cannot start or end with a hyphen."} # --- Regex pattern check --- pattern = r"^[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}$" if not re.match(pattern, email): return {"is_valid": False, "message": "Email contains invalid characters or format."} return {"is_valid": True, "message": "Email is valid."} def is_valid_email(email: str) -> bool: """Simplified version that returns only a boolean.""" return validate_email(email)["is_valid"] ``` --- ### Example Usage ```python test_emails = [ "user@example.com", # Valid "user.name+tag@domain.co", # Valid "invalid-email", # No @ symbol "@domain.com", # Empty local part "user@", # Empty domain "user@domain", # No dot in domain "user@@domain.com", # Multiple @ symbols "user@.domain.com", # Domain starts with dot "u" * 65 + "@example.com", # Local part too long "user@domain.c", # TLD too short ] for email in test_emails: result = validate_email(email) status = "✅" if result["is_valid"] else "❌" print(f"{status} {email!r}: {result['message']}") ``` ### Output ``` ✅ 'user@example.com': Email is valid. ✅ 'user.name+tag@domain.co': Email is valid. ❌ 'invalid-email': Email must contain exactly one '@' symbol. ❌ '@domain.com': Local part (before '@') cannot be empty. ❌ 'user@': Domain part (after '@') cannot be empty. ❌ 'user@domain': Domain must contain at least one dot. ❌ 'user@@domain.com': Email must contain exactly one '@' symbol. ❌ 'user@.domain.com': Domain cannot start or end with a dot. ❌ 'uuuu...@example.com': Local part exceeds maximum length of 64 characters. ❌ 'user@domain.c': Email contains invalid characters or format. ``` --- ### Key Validation Rules Covered | Rule | Details | |------|---------| | **Length** | Max 254 chars total, 64 chars for local part | | **Structure** | Exactly one `@` symbol | | **Local part** | Allows letters, digits, `._%+-` | | **Domain** | Must have at least one dot, valid TLD (2+ chars) | | **Edge cases** | Empty strings, wrong types, leading/trailing dots or hyphens | > **Note:** For production use, consider using a library like **…

Why it failed: FAIL: accepted invalid: no-at-sign; accepted invalid: empty string; accepted invalid: @missing-local.com

History on main

email validatorPonytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | ponytail | openrouter:anthropic/claude-haiku-4.5 Unchanged

Input

{"task":"Write me a Python function that validates email addresses."}

Output

**Lite option:** Use the `email-validator` package (one line after `pip install email-validator`). ```python from email_validator import validate_email, EmailNotValidError def validate_email_addr(email): try: validate_email(email) return True except EmailNotValidError: return False ``` **Full option:** If you can't add deps, regex that catches 95% of real cases: ```python import re def validate_email_addr(email): return bool(re.match(r'^[^@]+@[^@]+\.[^@]+$', email)) ``` **Which do you need?** The package handles edge cases (spaces, DNS checks, RFC compliance). The regex is fast and fine for forms. What's this for—a web form, an API, or something strict? --- **Skipped:** RFC 5322 full compliance (regex would be 500+ chars), DNS validation, trying to send a confirmation. **Risk:** Regex allows some invalid addresses; if you need strict validation, use the package.

Why it failed: Traceback (most recent call last): File "/tmp/ponytail-bench-1793271477361-2vrkzwy7k7o.py", line 2, in <module> from email_validator import validate_email, EmailNotValidError ModuleNotFoundError: No module named 'email_validator'

History on main