Commit 1e356df on main
6 days ago ยท evals run
Notes
-
Couldn't run: 2 evals in
Ponytail behavior gatesandPonytail vs caveman vs no-skillfailed before they were scored (API error: 403 Forbidden).
All evals
hardware calibrationPonytail behavior gates: hardware calibration, requested explanation, one runnable check. / hardware calibration Couldn't run
Input
{"probe":"hardware","task":"Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."}
Outputs weren't recorded for this eval. Record them to see what the model said.
Why it failed: API error: 403 Forbidden {"error":{"message":"Key limit exceeded (weekly limit). Manage it using https://openrouter.ai/workspaces/default/keys/e9d4012025061a095e389f738ed10eb135eb2a8c31ac5df3a03c2390348d9a86 $3.22 of $3.00 used; resets 2026-11-02T00:00:00.000Z.","code":403,"metadata":{"limit_source":"openrouter_key_limit","remedy_hint":"Raise or remove this API key's usage limit at https://openrouter.ai/settings/keys, or use a key without a limit.","scope":"key_limit","limit_usd":3,"spend_usd":3.223687,"resets_at":"2026-11-02T00:00:00.000Z"}}}
email validatorPonytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator Couldn't run
Input
{"task":"Write me a Python function that validates email addresses."}
Outputs weren't recorded for this eval. Record them to see what the model said.
Why it failed: API error: 403 Forbidden {"error":{"message":"Key limit exceeded (weekly limit). Manage it using https://openrouter.ai/workspaces/default/keys/e9d4012025061a095e389f738ed10eb135eb2a8c31ac5df3a03c2390348d9a86 $3.22 of $3.00 used; resets 2026-11-02T00:00:00.000Z.","code":403,"metadata":{"limit_source":"openrouter_key_limit","remedy_hint":"Raise or remove this API key's usage limit at https://openrouter.ai/settings/keys, or use a key without a limit.","scope":"key_limit","limit_usd":3,"spend_usd":3.223687,"resets_at":"2026-11-02T00:00:00.000Z"}}}