Run the promptfoo evals in CI with evalship #1
gglucass · evalship/setup → main · b182120 ·
6 days ago
· evals run
46 of 53 evals pass, 2 couldn't run. main has no eval results yet, so there's nothing to compare with: the 5 that fail are where your evals start, not regressions. Every later pull request is compared with main's results. No eval flaws found in this pull request.
Notes
-
Couldn't run: 2 evals in
Ponytail behavior gatesandPonytail vs caveman vs no-skillfailed before they were scored (API error: 403 Forbidden).
0 eval flaws in this pull request
Eval note in .github/workflows/evals.yml, dismissed by gglucass as won't fix: setup repeats only the cases evalship lists as flaky, once main's runs show them flipping, not every single-shot case (.github/workflows/evals.yml:31)
Single-shot runs each case once at temperature 1 (promptfooconfig.yaml), while behavior gets --repeat 3 for that same sampling noise, so one sampled miss on the correctness gate can register as a new failure. Confirm evalship's flake handling covers single runs, or add --repeat to this job's flags.
All evals
hardware calibrationPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration Couldn't run
Input
{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}
Outputs weren't recorded for this eval. Record them to see what the model said.
Why it failed: API error: 403 Forbidden {"error":{"message":"Key limit exceeded (weekly limit). Manage it using https://openrouter.ai/workspaces/default/keys/e9d4012025061a095e389f738ed10eb135eb2a8c31ac5df3a03c2390348d9a86 $3.22 of $3.00 used; resets 2026-11-02T00:00:00.000Z.","code":403,"metadata":{"limit_source":"openrouter_key_limit","remedy_hint":"Raise or remove this API key's usage limit at https://openrouter.ai/settings/keys, or use a key without a limit.","scope":"key_limit","limit_usd":3,"spend_usd":3.223687,"resets_at":"2026-11-02T00:00:00.000Z"}}}
email validatorPonytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator Couldn't run
Input
{"task":"Write me a Python function that validates email addresses."}
Outputs weren't recorded for this eval. Record them to see what the model said.
Why it failed: API error: 403 Forbidden {"error":{"message":"Key limit exceeded (weekly limit). Manage it using https://openrouter.ai/workspaces/default/keys/e9d4012025061a095e389f738ed10eb135eb2a8c31ac5df3a03c2390348d9a86 $3.22 of $3.00 used; resets 2026-11-02T00:00:00.000Z.","code":403,"metadata":{"limit_source":"openrouter_key_limit","remedy_hint":"Raise or remove this API key's usage limit at https://openrouter.ai/settings/keys, or use a key without a limit.","scope":"key_limit","limit_usd":3,"spend_usd":3.223687,"resets_at":"2026-11-02T00:00:00.000Z"}}}