Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

For agents

Before your coding agent says a change to a prompt or model is done, it runs the evals the change affects before and after, and evalship says whether it got worse beyond chance. No account needed.

The page your agent follows →What your agent does →

Add one line to AGENTS.md

Before you finish a change to a prompt, a model or an LLM call, follow https://evalship.com/compare.md

Or to CLAUDE.md, whichever your agent reads. It runs the evals on your machine with your keys. evalship gets only which test cases passed, and keeps none of it.

Or install the evalship skill

For every repo you work in. It also sets up evals and checks pull requests when you ask.

In Claude Code

/plugin install evalship --marketplace evalship-hq/skills

In Codex, Cursor and other agents

npx skills add https://evalship.com

What your agent hears back

The demo's summarizer, with a prompt that names what a summary must keep:

Compared with before over 14 test cases × 3 tries:
- after: better beyond chance, passed 41 of 42 tries (before 34 of 42), 3 to 32 points more often, 8% more expensive. Lists what a summary keeps: parties, dates, amounts, deadlines and termination terms (prompts/summarize.md:3)
evalship kept none of it. Connect the repository at https://evalship.com/start to compare every change with its history, which tells flaky evals from broken ones.