Skip to content

Optimize

Your coding agent finds changes your evals can prove better, runs them on your machine, and evalship says which win beyond chance.

Step by step

  1. Copy the line from your repository's Optimize page on evalship and paste it into your coding agent (Claude Code, Codex or another) in the repository. To try one kind of experiment only, such as Model, pick it under Optimize again, or name it after the line.
  2. The agent reads your LLM code and what evalship knows of it: your suites, how they did on the default branch, flaky evals, newer and cheaper models, and the experiments tried before. It proposes up to 5 changes, each with the eval that decides it and what a run costs. You pick one. A change that crosses options, such as memory layouts × models, can run as a screen of up to 31 variants.
  3. If no eval can tell the change apart from the product as it is, the agent writes one first, on a branch of its own. For retrieval and memory, it adds questions your model can't answer from what it already knows, so a variant can't pass on the model alone. For memory your product writes itself, the eval replays made-up sessions, asks about them later, and asks the same on a memory written by hand: you see whether learning or recall loses the facts.
  4. It runs the product as it is once to price a run, and asks before running more. It also says what a gap must be to matter: 5 points of pass rate, unless you pick another.
  5. It runs the product as it is, called current, and each variant against the same evals, 3 tries per test case unless you pick 2. A screen runs 2 tries each.
  6. It sends the results to evalship, which answers with a verdict per variant and what's next: a screen's best to confirm with fresh tries, more tries that would settle a close call and their price, or to stop. The Optimize page shows each experiment as the question it asked and why, evalship's answer, what it means for you and your users, and what's next.
  7. For a variant that wins, it offers a pull request with its change and evalship's verdict. When production also needs a setting, the pull request starts with a task for it. You merge it, or don't. The Optimize page shows the pull request with its experiment, and whether it merged.
  8. When you stop, it ends with what won, what's left for you, the candidates not tried and what it cost. Paste the same line to optimize again: it starts from the experiments before, and from the candidates you didn't pick.

What it tries

The agent starts from why your failing evals fail: it notes the first thing that went wrong in each, and starts from the biggest group. Each experiment changes one part of an AI call, for the same results for less or more evals passing. It goes beyond this list where your code points elsewhere, and changes your product's LLM features only.

Model

  • A newer model.
  • A cheaper model, evalship's suggestions first.
  • A lower reasoning effort, or none.
  • The sampling settings the provider recommends for the model, such as a low temperature where there's one right answer.
  • Hard inputs to a stronger model, easy ones to a cheaper one.
  • Where your users pick the model or bring their own key: each model they can choose, so you know which to recommend and which to warn about.

A swap changes how the prompt lands, so the agent adjusts it too. A cheaper model drops instructions the current one follows unasked. A newer one can overreact to emphatic rules.

Context

  • Fewer or shorter retrieved documents.
  • A denser format, such as a table instead of the same keys on every row.
  • Tools loaded only when a task needs them.
  • For a long agent run, its history compacted and old tool results cleared.
  • Tool results written like a prompt: short and plain, names rather than IDs.
  • What every call shares first and unchanged, so your provider's prompt caching applies.
  • Long documents first, each tagged with its source, and the question after them.
  • Keyword and embedding search together, where questions carry names or IDs.
  • A follow-up rewritten as a question that stands on its own.
  • Filters, such as dates, taken from the question.
  • How many documents retrieval returns, and how they're split.
  • How memory or docs are arranged and loaded: an index file, by topic, summaries or full text, up front or on demand.
  • How your product writes its own memory: when, what it keeps, whether it appends, merges or rewrites, and which model writes it.
  • Examples picked per input.

Instructions

  • Three to five varied worked examples of the inputs that fail, written for your product, never taken from your evals.
  • Each rule in one place with its reason, saying what to do rather than what to avoid.
  • What to do when a request is unclear or out of scope, or a tool fails.
  • For a classifier: each category defined, with examples where two overlap.
  • A short reasoning step for a multi-step task, or none for a simple one.
  • For agents: tool names, descriptions and arguments that make the right call obvious, and tool errors that name the fix.

Output

  • A schema or tool call instead of parsing free text, with a reasoning field first where the task needs thought.
  • Fewer output tokens, asked for or by a shorter shape. Writing them takes most of a call's time.
  • For long documents: the relevant quotes first, then the answer, with each claim's source next to it.

Flow

  • Calls merged, or an overloaded prompt split into steps that each do one thing.
  • Calls that don't depend on each other run at once.
  • A cache for a call whose input repeats exactly.
  • The majority of a few tries, where one reply is unreliable.
  • Code or a fixed reply where a call isn't needed.
  • A code tool where the model counts or calculates.
  • For agents: fewer, purpose-built tools.
  • For agents: a plan checked before it runs, and a cap on tool calls per task.
  • For agents: a subagent with a fresh context for a search that would flood the main one.

Checks

  • A second pass that checks the reply against the rules, then fixes or drops what fails.
  • Code for what the model gets wrong mechanically: units, number formats, broken JSON, duplicates.
  • "I don't know" when the context lacks the answer.
  • Cited sources that must be in the context.
  • For agents: the final claim checked against the tool results.
  • Your judges, guardrails and model warnings, tested on replies labeled good or bad.

Each change aimed at less is checked against evals that can fail a shallow reply.

What runs where

Everything runs on your machine with your API keys, and the agent asks before each run, with its price. evalship runs no model and none of your code for Optimize, and no GitHub Actions workflow runs. The agent sends the results with the upload token it gets when you approve it in the browser, as for evals you run on your machine.

How evalship decides

Each variant is compared with current over the test cases both ran. Every try counts: evalship pools them and runs a one-sided Fisher exact test each way, and a variant is better or worse when chance would give its pass count less than 5% of the time. The variants and both directions share that 5%, so an experiment of 7 variants crowns one by chance less than 1 time in 20. Pooling treats test cases of different difficulty as one, so it errs toward within chance.

Where several checks grade one reply, such as a recipe's steps, times and servings, they rise and fall together: counted as separate tries, they would make the test far surer than it is. So when each try names its reply, evalship counts replies instead, and a reply passes only when all its checks pass.

Tries that couldn't run (a rate limit, an outage, a missing key) are left out, as they are in a pull request's comparison. Cost, tokens and time are compared per try, over the test cases where both recorded them, and aren't tested for chance.

Each verdict comes with the range the gap could be, in points of pass rate, such as "between 6 points less often and 2 more often". It's Newcombe's interval from two Wilson intervals at the test's own bar, kept on the test's side of zero.

When it's within chance

A gap matters at the experiment's margin: 5 points of pass rate, unless the agent sends another. A variant within chance that's at least 10% cheaper, lighter or faster counts as good as current when its range shows it passes at most the margin less often. That's the kind of win the agent offers a pull request for. A variant whose range can't reach the margin isn't worth more tries.

When the range still leaves room either way, evalship says how many more tries per test case would settle it, 80% of the time if the gap is what it looks like, and what they cost. They're pooled with the first tries and judged at half the bar, so two looks crown a variant by chance less than 7.5% of the time. evalship allows one more look per experiment. When settling it would cost more than 3 times the experiment, it recommends stopping, with the price: settling a cheaper model can still pay for itself in production. It never asks to run until the gap turns out beyond chance: looking often enough would crown a variant by luck.

Screens

A screen runs up to 31 variants on few tries. With that many, the best-looking gaps are partly luck, so a screen crowns none. evalship names up to 3 to confirm against current with fresh tries, where gaps count as above: the one that saves the most while within the margin, then those that passed most often.

A variant marked as a reference, such as a memory written by hand, shows what your product could reach. evalship compares it like any other, and never offers to ship it.

What it never does

  • Change an eval, its checks or its data to make a variant win, or put an eval's expected answers into your prompts, examples, memory or documents.
  • Run more than you approved.
  • Touch your default branch or your working tree: variants live in git worktrees.
  • Merge anything.
  • Read your secrets, or copy customer data into test cases.

The results file

The agent writes evalship JSON (Result formats) with the variants and their tries, "schema": "evalship/experiment-v1":

{
  "schema": "evalship/experiment-v1",
  "title": "Cheaper model for support replies",
  "question": "Can support replies run on Haiku for less, and stay as good?",
  "why": "Every support reply runs on Sonnet (app/support_bot.py:12), and the support-bot suite passes 46 of 48 on main.",
  "goal": "The same pass rate for less cost",
  "variants": [
    { "name": "current", "change": "As it is" },
    { "name": "haiku", "change": "claude-haiku-5-5 instead of claude-sonnet-5-5 (app/support_bot.py:12)", "part": "model" }
  ],
  "evals": [
    { "variant": "current", "suite": "support-bot", "key": "refund over the limit", "passed": true, "tokens": 1840, "cost_usd": 0.0031 },
    { "variant": "haiku", "suite": "support-bot", "key": "refund over the limit", "passed": false, "tokens": 1790, "cost_usd": 0.0009 }
  ]
}

current comes first. Each try is one entry, and every variant runs the same test cases. Add "margin" for a margin other than 5 points, "screen": true for a screen, "follows" with an experiment's branch for its next step, and "reference": true on a variant that isn't a change to ship. An experiment that crosses models with other changes gives every variant its model, and the page shows them as a table of models by changes. Send it with an upload token:

curl --fail-with-body -sS https://evalship.com/api/v1/optimize \
  -H "Authorization: Bearer $EVALSHIP_TOKEN" \
  -F repository=OWNER/REPO \
  -F file=@evalship-optimize.json

evalship keeps the title, the question, why the agent tried it, the goal, each variant's name, one-line change, model and the part of the call it changes, and each test case's name with how often it passed and its tokens, cost and time. It keeps no inputs, replies or code. See Security.