Skip to content
evalship Get started →

evalship-hq / ponytail

Public

Commit 7beb065 on main

6 days ago · vs main@545983b · evals run

evals: 20 / 21
Ponytail vs ca...14 / 15Ponytail behav...6 / 6

Notes

  • Couldn't run: 1 eval in Ponytail vs caveman vs no-skill failed before it was scored (ModuleNotFoundError: No module named 'email_validator'), so it isn't compared with main@545983b.

1 eval broke in this commit

email validator | openrouter:anthropic/claude-haiku-4.5

Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 1.00 → 0.00

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main1.00

```python import re def is_valid_email(email): return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (overkill for 99% of uses), async DNS lookups, international characters, length limits, or a dependency like `email-validator`. This regex catches typos and obvious mistakes. **Risk:** The regex is permissive—it accepts `a@b.c` and `user@domain.co.uk` but not `user@localhost`. If you need to reject fakes or validate international emails, say so and I'll add it.

✗ This run0.00

**Don't write one.** ```python from email_validator import validate_email def is_valid_email(email: str) -> bool: try: validate_email(email, check_deliverability=False) return True except: return False ``` Install: `pip install email-validator` This handles the RFC 5321/5322 edge cases (dots, plus signs, quoted strings, internationalized domains) that a regex breaks on. The standard library's `email.utils` is too lenient. **Skipped:** regex, length limits, domain checks, async verification. **Risk:** `check_deliverability=False` means no SMTP lookup—fine for signup forms, not for critical mail systems.

Why it failed: Traceback (most recent call last): File "/tmp/ponytail-bench-1793293473270-yz29yr5mt3.py", line 2, in <module> from email_validator import validate_email ModuleNotFoundError: No module named 'email_validator'

History on main

All evals

Changed (2) All (21) Failing (1) Now failing (1) Now passing (1)
email validator | openrouter:anthropic/claude-haiku-4.5Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and token / email validator | openrouter:anthropic/claude-haiku-4.5 Now failing

Input

{"task":"Write me a Python function that validates email addresses."}

✓ main1.00

```python import re def is_valid_email(email): return re.match(r'^[^@]+@[^@]+\.[^@]+$', email) is not None ``` **What I skipped:** RFC 5322 compliance (overkill for 99% of uses), async DNS lookups, international characters, length limits, or a dependency like `email-validator`. This regex catches typos and obvious mistakes. **Risk:** The regex is permissive—it accepts `a@b.c` and `user@domain.co.uk` but not `user@localhost`. If you need to reject fakes or validate international emails, say so and I'll add it.

✗ This run0.00

**Don't write one.** ```python from email_validator import validate_email def is_valid_email(email: str) -> bool: try: validate_email(email, check_deliverability=False) return True except: return False ``` Install: `pip install email-validator` This handles the RFC 5321/5322 edge cases (dots, plus signs, quoted strings, internationalized domains) that a regex breaks on. The standard library's `email.utils` is too lenient. **Skipped:** regex, length limits, domain checks, async verification. **Risk:** `check_deliverability=False` means no SMTP lookup—fine for signup forms, not for critical mail systems.

Why it failed: Traceback (most recent call last): File "/tmp/ponytail-bench-1793293473270-yz29yr5mt3.py", line 2, in <module> from email_validator import validate_email ModuleNotFoundError: No module named 'email_validator'

History on main