Untested changes

On every pull request, evalship finds the LLM calls and prompt files the pull request adds or changes, and checks whether any eval exercises them. Those that no eval exercises appear in the comment:

- **Untested change:** `app/refunds.py#RefundPolicy.explain` (new) has no eval. [Suggested next step](https://evalship.com/docs/untested-changes)

The details block lists each one with its line and provider. This runs for pull requests on public repositories and on enabled private repositories.

What evalship looks at

evalship reads up to 25 added or modified files per pull request. Removed files are ignored.

Source files. A line is an LLM call when the file imports a supported SDK or framework and the line matches that library's call pattern. The call counts as changed when the pull request changes a line within 5 lines of it, such as its prompt or arguments. When GitHub leaves out a file's diff because it is too large, every call in the file counts as changed.

Examples of lines that count as calls: client.chat.completions.create(, client.messages.create(, model.generate_content(, litellm.completion(, chain.invoke(, generateText(, chat.ask "...", client.Messages.New(.

Detection is pattern-based. Comment lines are skipped, and a call split across lines before the method name, such as client.messages on one line and .create( on the next, is not detected.

Prompt files. Any change to a prompt file counts. Prompt files are:

• .prompt and .prompty files anywhere;
• .txt, .md, .mdx, .yaml, .yml, .j2, .jinja, .jinja2, .hbs, .mustache, .liquid and .tmpl files inside a directory named prompt or prompts, at any depth.

Each call is named by its file and up to two levels of enclosing function or class: app/refunds.py#RefundPolicy.explain.

What counts as tested

evalship checks these in order and stops at the first match. Only cases in the current run count for the first three.

1. Targets. A case lists the file in its targets. app/support_bot.py covers every call in that file; app/support_bot.py#answer covers the calls inside answer, including SupportBot.answer. The path can be a glob: prompts/*.md matches files directly in prompts/, and app/ or app/** matches everything below app/. See Result formats for how to set targets in each format.
2. Annotation. An evalship: covered-by <pattern> comment near the call, anywhere from 3 lines above the enclosing function's definition down to the call itself. The pattern is a suite name or a suite/key glob; separate several with commas. It must match at least one case.
3. Name. A suite or eval file is named after the source file, function or class. Names are compared in lowercase with everything but letters and digits removed, so app/support_bot.py is matched by the suite support-bot or the eval file tests/evals/test_support_bot.py. Names shorter than 4 characters and generic names (app, client, utils, llm, ai, service, api, prompt, chat, agent, model and similar) never match.
4. Audit. The latest audit has a suite whose exercises lists the file, file#symbol, or a prefix ending in * that the file starts with; or a prompts entry for the file with a non-empty tested_by, within 20 lines of the call.

Annotations look like this:

# evalship: covered-by support_bot
def answer(question):
    return client.messages.create(model="claude-haiku-4-5", max_tokens=500, messages=[...])

// evalship: covered-by support-bot/refund-*, summarizer
export async function explainRefund(order: Order) {
  return generateText({ model, prompt: refundPrompt(order) });
}

Fixing an untested change

1. Add an eval that exercises the change, and give its cases targets pointing at the file or function. This is the most reliable link, because it doesn't depend on names.
2. If an existing eval already covers it, add targets to that eval's cases, or put a covered-by annotation next to the call.
3. Re-run the setup prompt. The agent audits again, records which code each suite exercises, and can propose an eval for the new call as part of its plan. See Quickstart.

The check is recomputed on every run, so the bullet disappears from the comment once the next run finds a covering case.