Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

About evalship

evalship catches issues in your AI prompts and responses before your users do. It works next to the evals and tools you already have.

Why it exists

Evals are meant to catch an LLM feature getting worse before your users do. Often they don't. The judge counts errors as passes. No test case checks that your support bot only promises the refunds your system allows. A pull request rewrites a prompt that nothing covers. One flaky run fails, and the team learns to ignore red.

Customers hold you to what the bot says. Air Canada's chatbot told a customer they could claim a bereavement fare after flying, which the airline's policy didn't allow. A tribunal made Air Canada pay the difference: Moffatt v. Air Canada, 2024. AI is never wrong logs more.

A score of 0.92 on a dashboard shows none of that. evalship checks the evals themselves, then keeps watch on every pull request. In production, it checks every AI call and emails you when one goes wrong.

Our goal is to put your AI calls on autopilot.

What it does

Test

  • At setup, your coding agent reviews your evals and finds their flaws.
  • A pull request that changes a prompt, an AI call or an eval gets one comment: the evals it broke, apart from the flaky ones, and what no eval covers.
  • Your coding agent checks a change the same way before it finishes.

Monitor

  • Your app sends evalship its AI calls in production, and evalship checks each one.
  • What went wrong is grouped into issues on your To do list.
  • You get an email when an issue opens.

Optimize

  • Your coding agent tries prompt and model changes on your machine, such as a newer or cheaper model.
  • evalship says which win beyond chance, and by how much.
  • A winner becomes a pull request you merge, or don't.

What it doesn't do

evalship doesn't run your evals: your workflow does, or your coding agent on your machine, with your API keys. It doesn't store datasets or manage prompts. It doesn't need an eval platform either. If you use Langfuse, Braintrust or LangSmith, keep it: each eval can link to its trace.

Who's behind it

evalship is made by Garm Lucassen at Garm Tech B.V., a software company in Amsterdam. Questions, ideas or a bug: hello@evalship.com.

Garm Tech B.V., Staalkade 6 O, 1011 JN Amsterdam, the Netherlands. Chamber of Commerce 99602075, VAT NL869056748B01.

Set up https://evalship.com/setup.md