Result formats

evalship reads six result formats. Each result becomes the same eval record: a suite, a key, pass or fail, and optionally a score, threshold, duration, cost, tokens, input, output, failure message, file, targets, url and the AI calls behind the reply.

Auto-detection

evalship detects the format from the content, not the file name:

1. Content that starts with < is JUnit XML.
2. A zip file is an Inspect AI .eval log.
3. A JSON object with an evals array is evalship JSON.
4. A JSON object with evalId or a results.results array is promptfoo JSON.
5. A JSON object with a tests array whose entries have a nodeid is pytest-json-report JSON.
6. A JSON object with a testResults array, with numTotalTests or with assertionResults in its entries, is Vitest or Jest JSON.
7. A JSON object with an eval object and a version is an Inspect AI JSON log.

Anything else is skipped, so other .xml, .json and .eval files in the same artifact do no harm. A leading byte order mark is ignored.

JUnit XML

Each <testcase> is one eval. Evals with a <skipped> child are ignored.

The stem of a file is its base name up to the first dot, without a test_, eval_ or evals_ prefix and without a _test, _spec, _eval or _evals suffix. tests/evals/test_support_bot.py has the stem support_bot.

For example, pytest writes a testsuite named pytest and this testcase:

<testcase classname="tests.evals.test_support_bot" name="test_refund_limit"
          file="tests/evals/test_support_bot.py" time="1.84">

evalship reads it as suite support_bot, key tests.evals.test_support_bot.test_refund_limit.

Properties

A <properties> block inside a <testcase> adds data to the eval:

In JUnit, a score below its threshold does not fail an eval by itself; pass or fail always comes from <failure> and <error>.

<testcase classname="tests.evals.test_support_bot" name="test_refund_limit"
          file="tests/evals/test_support_bot.py" time="1.84">
  <properties>
    <property name="score" value="0.41"/>
    <property name="threshold" value="0.7"/>
    <property name="targets" value="app/support_bot.py#answer,prompts/support_system.md"/>
  </properties>
  <failure message="assert 0.41 &gt;= 0.7">...</failure>
</testcase>

pytest: record_property

pytest's record_property fixture writes properties into the XML:

def test_refund_limit(record_property):
    question = "Can I get a refund of $500?"
    reply = answer(question)
    score = judge_refund_policy(reply)
    record_property("score", score)
    record_property("threshold", 0.7)
    record_property("targets", "app/support_bot.py#answer")
    record_property("input", question)
    record_property("output", reply)
    assert score >= 0.7

Run pytest with -o junit_family=xunit1:

pytest tests/evals --junitxml=eval-results.xml -o junit_family=xunit1

pytest's default family, xunit2, doesn't write per-test properties. To set it once, add this to pyproject.toml:

[tool.pytest.ini_options]
junit_family = "xunit1"

promptfoo JSON

Each test, prompt and provider combination is one eval.

A test without a description gets the key test <8 hex characters>, derived from its vars. Changing any var then changes the key.

With two prompts and two providers, a key looks like Refund above limit | prompts/support_v2.txt | openai:gpt-4o-mini.

pytest-json-report JSON

Each test is one eval, keyed the way pytest's JUnit XML keys it, so switching between the two keeps every eval's history. Skipped and xfailed tests are ignored.

user_properties, which pytest's record_property writes, work like JUnit properties: score, threshold, targets, input, output, cost_usd, tokens and url, and any other name goes into metadata.

Vitest or Jest JSON

Each test is one eval, keyed the way Vitest's JUnit XML keys it, so switching between the two keeps every eval's history. Skipped, pending and todo tests are ignored.

Inspect AI logs

Each sample is one eval, and one per scorer when the task has more than one. Upload the .eval files from Inspect's logs/ folder as they are.

A task run with --epochs gives each sample several results with the same key and input. evalship reads them as one eval with repeats (Stable keys): it passes when at least half of them pass.

evalship JSON

The schema is at evalship.com/schema/v1.json.

{
  "schema": "evalship/v1",
  "suite": "support-bot",
  "url": "https://www.braintrust.dev/app/acme/p/support-bot/experiments/pr-142",
  "evals": [
    {
      "key": "refund-above-limit",
      "name": "Declines a refund above $100",
      "passed": true,
      "score": 0.92,
      "threshold": 0.7,
      "duration_ms": 1840,
      "cost_usd": 0.0021,
      "tokens": 812,
      "input": "Can I get a refund of $500?",
      "output": "I can refund up to $100. For larger amounts I'll pass you to a colleague.",
      "file": "evals/support_bot.eval.ts",
      "targets": ["app/support_bot.py#answer", "prompts/support_system.md"],
      "url": "https://cloud.langfuse.com/project/acme/traces/4f1c9e",
      "metadata": { "model": "gpt-4o-mini" }
    }
  ]
}

The top-level suite and url apply to evals without their own. Only evals is required at the top level.

Non-string values in input, output and failure are serialized as JSON before truncation.

calls

The AI calls an eval's reply took, in order, so its report shows what each model got and answered. Each is an object with call, its name, and optionally kind (ai, tool or step), model, input, output, error, input_tokens, output_tokens, cost in dollars and duration_ms. The first 20 calls that fit in 64 KB are kept, each input and output truncated to 16 KB and each error to 4 KB. tinyeval records them from TINYEVAL_CALLS.

An app that already sends OpenTelemetry traces can send them instead. In the eval job, set the OTEL_EXPORTER_OTLP_TRACES_* variables the repo's Monitor page gives when you connect production, and tag each eval's spans with evalship.eval, a resource or span attribute, set to the eval's otel. tinyeval does this for each run of ask when the endpoint is set. An eval with no calls of its own gets the AI call spans with its id that arrived before its results. They don't show on Monitor or count toward its spans. evalship deletes them the night after they arrive.

"calls": [
  { "call": "classify-ticket", "model": "claude-haiku-5-5", "input": "Can I get a refund of $500?", "output": "refund", "cost": 0.0001 },
  { "call": "support-chat", "model": "claude-opus-5-5", "input": "...", "output": "I can refund up to $100.", "input_tokens": 1840, "output_tokens": 61 }
]

url

If your evals also report to Langfuse, Braintrust, LangSmith or another platform, give each eval a url that opens it there, such as its trace. The PR comment links each changed eval's name to it, and the report shows a "View on" link next to each eval. Only http and https links are kept, up to 2,000 characters. Integrations has the code for each platform.

Stable keys

evalship matches evals across runs by suite and key. When either changes, the old eval shows up as removed and the new one as new, and its flakiness history starts over.

• Keep timestamps, random ids, dates and changing model names out of test names and keys.
• In pytest, give parametrized tests explicit ids= so parameter values don't end up in the key.
• In JUnit, the key includes the module path, so moving or renaming a test file changes its keys.
• In promptfoo, give every test a description.
• If the same suite and key appear twice in one file with the same input, they're one eval with repeats (see How the PR comment works). With different inputs, the later ones become key (2), key (3) in file order. That depends on order, so give each eval its own key.

A workflow run's result files are read until they hold 20,000 evals; the files after that are skipped. A file with more than 20,000 evals is skipped whole, not cut. See Connecting CI for artifact and file size limits.