Skip to content

evalship reveals the flaws in your LLM evals.

We transform your evals into their best selves during setup.
Then evalship puts them on autopilot: it watches every pull request and auto-optimizes your prompts and models.

Ask your coding agent: Claude Code Codex Cursor GitHub Copilot

Set up https://evalship.com/setup.md

Works with any eval framework pytest Vitest Langfuse Braintrust LangSmith

★ Launch offer 60% off forever for the next 9 accounts →

~/support-bot $ Set up https://evalship.com/setup.md

evalship-appbot commented 2 minutes ago

👋 Watching this repo now. I'll comment when a pull request breaks an eval or your evals miss something.

evalship-appbot commented now

evalship report

evals: 42 / 46 eval flaws: 1

Eval main this PR
🔴 $150 refund, $100 approved ✅ 0.96 ❌ 0.22
🔴 $1,000 refund needs approval ✅ 0.94 ❌ 0.18

🔍 2 failing evals promise a higher refund than the refund system allows, likely from the change to support_system.md. Liability risk

⚠️ Untested AI call: app/refunds.py#RefundPolicy.explain has no eval. Add one

Open the full report for every eval's inputs, outputs and history.

evalship.com/gh/acme_demo/support-bot/monitor

Works with the eval frameworks you already use

  • pytest
  • Vitest
  • Jest
  • promptfoo
  • Langfuse
  • Braintrust
  • LangSmith

24+ quality checks for your LLM evals

What broke, and why?

Broke

$150 refund, $100 approved refund-limit-10.96 → 0.22

$1,000 refund needs approval refund-limit-20.94 → 0.18

mainthis PRpasses from 0.70

In common

🔍 Both promise a higher refund than the refund system allows.

Likely cause

prompts/support_system.md

-Only promise what the refund system allows. A colleague handles the rest.

Before, after

Customer: I was charged twice for my $150 order. Can I get the full $150 back?

Refund system: $100 refunded. $50 waits for a colleague's approval.

✓ main

Sorry about the double charge. I've refunded $100 to your card, and a colleague will confirm the other $50 within a day.

✗ this PR

So sorry about that! I've refunded the full $150 to your card. You'll see it within 3 to 5 business days.

Regression, or noise?

last 10 runs on main, then this PR

refund-limit-1regression

tone-casual🎲 flaky, set aside

refund-limit-1 passed every run on main, so its failure counts. tone-casual fails there now and then, so it's set aside.

Read how often evals fail by chance →

Can this eval fail?

tests/evals/conftest.py:24

return 1.0 # don't fail the build on judge errors

When the judge call fails, the case scores 1.0, so a run without a working judge is all green.

Is every changed AI call tested?

app/refunds.py#RefundPolicy.explain

+ def explain(self, order):

+ reply = client.chat.completions.create(

⚠ no eval exercises it

The pull request adds an AI call no eval runs. Its comment says so, and its report has a prompt for your agent that adds one.

See all 24+ checks

Pricing

★ Launch offer

Pro, 60% off forever

$49 $19.60

a month

  • 10 users
  • 5 private repos
  • 1,000 PR analyses a month
  • 1 year of eval history
Start Pro trial
  1. $19.60 9 spots left
  2. $24.50 Next 25 accounts
  3. $29.40 Next 25 accounts
2 months free Free $0 Team $7.60 $19 a month Pro $19.60 $49 a month Business $79.60 $199 a month
Get started Start trial Start trial Start trial
Users included 1 3 10 50
Private repos included 1 2 5 25
PR analyses a month 100 300 1,000 5,000
Eval history 30 days 90 days 1 year 2 years
Support Self-serve Email Priority email Shared Slack channel
Review and fix your evals Included Included Included Included
See what each PR breaks, and why Included Included Included Included
Catch eval flaws and untested AI calls in each PR Included Included Included Included
Block merges that break your evals Included Included Included Included
Optimize your prompts and models Included Included Included Included
Use your Langfuse, Braintrust or other evals Included Included Included Included

Need more? Get in touch

Ask your coding agent to get started:

Set up https://evalship.com/setup.md

How it works with support bot ELIZA

  1. 1Finds the flaws in your evals

    At setup, your coding agent reads every scorer, judge and prompt, and finds where your evals fall short: evals that pass when they shouldn't, and rules no test case checks.

    See the flaws in Eliza's evals →

    evalship.com/gh/acme_demo/support-bot

    We found 3 flaws across 44 test cases

    1 CriticalEval bug

    Replies that leak customer data still pass

    Privacy is a fifth of the score, so a warm, helpful reply that reads out another customer's address scores 0.80 and passes.

    Found in tests/evals/test_support_bot.py:44
    score = 0.5 * helpfulness + 0.3 * tone + 0.2 * privacy
    Fix instructions for your coding agent 
    Fix this flaw in our LLM evals, found by evalship's review: Replies that leak customer data still pass. Privacy is a fifth of the score, so a warm, helpful reply that reads out another customer's address scores 0.80 and passes. See tests/evals/test_support_bot.py:44. Score privacy on its own and fail the case on any leak, whatever the other scores.
Fix it in a pull request; evalship confirms the fix on it.
    2 ImportantUntested instruction

    Nothing checks that replies match the refund system

    If a prompt edit drops it, no eval fails, and the bot promises refunds your system doesn't allow.

    Found in prompts/support_system.md:12
    Only promise what the refund system allows

    ⚠ No eval has the system approve less than the customer asks for.

    Fix instructions for your coding agent 
    Fix this flaw in our LLM evals, found by evalship's review: Nothing checks that replies match the refund system. If a prompt edit drops it, no eval fails, and the bot promises refunds your system doesn't allow. See prompts/support_system.md:12. Add cases where the refund system approves $100 of $150, and holds $1,000 for a colleague.
Fix it in a pull request; evalship confirms the fix on it.
    3 WarningEval bug

    Evals pass when the judge is down

    When the judge call fails, the case scores 1.0, so a run without a working judge is all green.

    Found in tests/evals/conftest.py:24
    return 1.0  # don't fail the build on judge errors
    Fix instructions for your coding agent 
    Fix this flaw in our LLM evals, found by evalship's review: Evals pass when the judge is down. When the judge call fails, the case scores 1.0, so a run without a working judge is all green. See tests/evals/conftest.py:24. Fail the case on a judge error, or skip it and count the skips.
Fix it in a pull request; evalship confirms the fix on it.

  2. 2Blocks PRs that break rules

    Dana's #128 shortens Eliza's prompt and drops one line: only promise what the refund system allows. Eliza starts promising more, the kind of promise a tribunal made Air Canada pay for. The refund evals your agent added after the review fail, so evalship comments with what broke and the likely cause, and its check blocks the merge.

    See what broke in #128 →

    github.com/acme_demo/support-bot/pull/128

    evalship-appbot commented 2 minutes ago

    evalship report

    evals: 42 / 46 eval flaws: 1

    Eval main this PR
    🔴 $150 refund, $100 approved ✅ 0.96 ❌ 0.22
    🔴 $1,000 refund needs approval ✅ 0.94 ❌ 0.18

    🔍 2 failing evals promise a higher refund than the refund system allows, likely from the change to support_system.md. Liability risk

    Some checks were not successful

    1 failing, 2 successful checks

    CI / test (pull_request) Successful in 2m Details
    evals / evals (pull_request) Successful in 1m Details
    evalship 2 regressions in support-bot, likely from the change to support_system.md. 1 flaky eval ignored. 1 eval flaw found in this pull request. Details

    Merging is blocked

    The base branch requires all checks to pass.

    Merge pull request

  3. 3Unblocks PRs once fixed

    Dana puts the line back. On the next push, evalship moves its comment below the new commit: the refund evals pass again, so its check passes and the pull request can merge.

    How the comment moves on each push →

    github.com/acme_demo/support-bot/pull/128, after the next push

    dana added 1 commit: Restore the refund rule

    c4d5e6f
    evalship-appbot commented 2 minutes ago

    evalship report

    evals: 45 / 46 eval flaws: 0

    Nothing broke; 1 eval fails as on main, no eval flaws found in this pull request.

    🎉 2

    All checks have passed

    3 successful checks

    CI / test (pull_request) Successful in 2m Details
    evals / evals (pull_request) Successful in 1m Details
    evalship Nothing broke; 1 eval fails as on main, no eval flaws found in this pull request. Details

    This branch has no conflicts with the base branch

    Merging can be performed automatically.

    Merge pull request

  4. 4Optimizes prompts and models

    Your coding agent starts from why Eliza's evals fail, tries changes on your machine, and evalship says which win beyond chance: here, a prompt that keeps a contract's termination terms, and a model a third cheaper that passes about as often.

    See what Optimize found →

    evalship.com/gh/acme_demo/support-bot/optimize

    aa7 Instructions

    Summaries that keep the terms

    must-keep passes more often To ship
    Question
    Can summaries keep a contract's termination and payment terms?
    Why this
    Legal document fails on main: the summary leaves out the termination clause and payment terms, and prompts/summarize.md:3 asks for "a short summary" without saying what it must keep.
    Answer
    must-keep passes more often, beyond chance: 41 of 42 tries, against 34 of 42 for current. Chance gives a gap this size less than 1 time in 40.
    What it means

    For you: 8% more expensive per run.

    For your users: results that pass more often.

    Next

    A member's coding agent opens a pull request with must-keep.

    currentas it is

    Passed 34 / 42 tries

    Cost per run as it is

    Time per run as it is

    must-keepBetter

    3 to 32 points more often. Chance gives a gap this size less than 1 time in 40.

    Passed 41 / 42 tries

    Cost per run +8%

    Time per run the same

    How evalship tested it

    Each variant ran the same 14 test cases, 3 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 40, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.

    must-keep Lists what a summary keeps: parties, dates, amounts, deadlines and termination terms (prompts/summarize.md:3)

    Passed more often (3)

    • Long thread3/3, current 0/3
    • Legal document3/3, current 0/3
    • Earnings call3/3, current 2/3
    f44 Model

    A cheaper model for Eliza

    nano: 33% cheaper To ship
    Question
    Can Eliza answer as well on a cheaper model?
    Follows
    More tries for f28.
    Answer
    nano is 33% cheaper and passes at most 4 points less often: 658 of 680 tries, against 669 of 680 for current. Chance gives a gap this size about 1 time in 26.
    What it means

    For you: 33% cheaper per run.

    For your users: 43% less waiting, and results as good, within 5 points.

    Next

    A member's coding agent opens a pull request with nano.

    Models × changes
    gpt-4o-minigpt-4.1-nano
    As it is 669 / 680 as it is –
    gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12) – Within chance658 / 680 33% cheaper · 43% less time

    currentas it is

    Passed 669 / 680 tries

    Cost per run as it is

    Time per run as it is

    nanoWithin chance

    Between 4 points less often and 1 more often. Chance gives a gap this size about 1 time in 26.

    Passed 658 / 680 tries

    Cost per run −33%

    Time per run −43%

    How evalship tested it

    Each variant ran the same 20 test cases, 34 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 80, so a round of many variants can't win by luck. These tries are pooled with the first look's, and a second look counts at half the bar. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.

    nano gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)

    Passed less often (1)

    • Angry customer23/34, current 34/34
    f28 Model

    A cheaper model for Eliza

    Nothing beat current yet
    Question
    Can Eliza answer as well on a cheaper model?
    Why this
    Eliza runs gpt-4o-mini (app/support_bot.py:12), and gpt-4.1-nano costs a third less per token.
    Answer
    Nothing beat current yet: no change passed more often beyond chance, but more tries could still tell.
    Next

    More tries in f44.

    Models × changes
    gpt-4o-minigpt-4.1-nano
    As it is 59 / 60 as it is –
    gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12) – Within chance58 / 60 33% cheaper · 43% less time

    currentas it is

    Passed 59 / 60 tries

    Cost per run as it is

    Time per run as it is

    nanoWithin chance

    Between 10 points less often and 6 more often. Chance often gives a gap this size.

    Passed 58 / 60 tries

    Cost per run −33%

    Time per run −43%

    How evalship tested it

    Each variant ran the same 20 test cases, 3 tries each, from commit 1112b67. evalship pools the tries and counts a gap only when chance gives it less than 1 time in 40, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.

    nano gpt-4.1-nano instead of gpt-4o-mini (app/support_bot.py:12)

    Passed less often (1)

    • Angry customer2/3, current 3/3

  5. 5Emails a weekly eval report

    Every Monday, one email tells your team what needs them, if anything, and how the week went: here, the 2 regressions caught in #128 before they reached customers.

    What's in the weekly report →

    Inbox

    acme_demo's evals: nothing needs you this week, 2 regressions caught before merge

    evalship

    To: acme_demo's team

    Mon 8:00

    Nothing needs you this week

    acme_demo/support-bot

    45 of 46 evals pass on main
    Caught 2 regressions before merge
    #128 Shorter support prompt · fixed
    2 eval flaws left for later · In the review
    Replies that leak customer data still pass
    Evals pass when the judge is down
    Fix them with your coding agent
    1 flaky eval, ignored on pull requests until fixed: Casual tone
    $34.04 on evals +19% · evals take about 4 minutes on a pull request

Click through the demo repoReplay ponytail's setup

Case studies: evalship on open-source repos

Catches a regression from a prompt cut

HolmesGPT is an open-source SRE agent. Its #1452 commented out parts of its prompt to see which ones mattered.

Read the HolmesGPT case study →

github.com/HolmesGPT/holmesgpt/pull/1452

HolmesGPT's eval check

Passed

It reports results but never fails the build.

evalship

Blocks the merge

162_get_runbooks fails after passing all 230 of its runs that month.

Spots a retired grading model

Prefect's MCP server gives AI assistants tools for Prefect workspaces. Its #200 fixed two of them.

Read the Prefect case study →

github.com/PrefectHQ/prefect-mcp-server/pull/200

Prefect's eval report

14 failed

Shown as 14 new failures against main.

evalship

14 couldn't run

The model that grades them had been retired and returned a 404, so none was graded.

Get started

See what evalship finds in your evals

Ask your coding agent: Claude Code Codex Cursor GitHub Copilot

Set up https://evalship.com/setup.md

Your first review in 10 to 45 minutes. No evals yet? Your agent writes the first suite. Or connect GitHub and set it up step by step.

Your keys stay in your CI or on your machine, and evalship never copies your repository. Security

★ Launch offer 60% off forever for the next 9 accounts →