Result formats

evalship reads three result formats. Each becomes the same case record: a suite, a key, pass or fail, and optionally a score, threshold, duration, cost, tokens, input, output, failure message, file and targets.

Auto-detection

evalship detects the format from the content, not the file name:

1. Content that starts with < is JUnit XML.
2. A JSON object with a cases array is evalship JSON.
3. A JSON object with evalId or a results.results array is promptfoo JSON.

Anything else is skipped, so other .xml and .json files in the same artifact do no harm. A leading byte order mark is ignored.

JUnit XML

Each <testcase> is one case. Cases with a <skipped> child are ignored.

The stem of a file is its base name up to the first dot, without a test_, eval_ or evals_ prefix and without a _test, _spec, _eval or _evals suffix. tests/evals/test_support_bot.py has the stem support_bot.

For example, pytest writes a testsuite named pytest and this testcase:

<testcase classname="tests.evals.test_support_bot" name="test_refund_limit"
          file="tests/evals/test_support_bot.py" time="1.84">

evalship reads it as suite support_bot, key tests.evals.test_support_bot.test_refund_limit.

Properties

A <properties> block inside a <testcase> adds data to the case:

In JUnit, pass or fail always comes from <failure> and <error>. A score below its threshold does not fail a case by itself.

<testcase classname="tests.evals.test_support_bot" name="test_refund_limit"
          file="tests/evals/test_support_bot.py" time="1.84">
  <properties>
    <property name="score" value="0.41"/>
    <property name="threshold" value="0.7"/>
    <property name="targets" value="app/support_bot.py#answer,prompts/support_system.md"/>
  </properties>
  <failure message="assert 0.41 &gt;= 0.7">...</failure>
</testcase>

pytest: record_property

pytest's record_property fixture writes properties into the XML:

def test_refund_limit(record_property):
    question = "Can I get a refund of 500 euros?"
    reply = answer(question)
    score = judge_refund_policy(reply)
    record_property("score", score)
    record_property("threshold", 0.7)
    record_property("targets", "app/support_bot.py#answer")
    record_property("input", question)
    record_property("output", reply)
    assert score >= 0.7

Run pytest with -o junit_family=xunit1:

pytest tests/evals --junitxml=eval-results.xml -o junit_family=xunit1

pytest's default family, xunit2, doesn't write per-test properties. To set it once, add this to pyproject.toml:

[tool.pytest.ini_options]
junit_family = "xunit1"

promptfoo JSON

Write it with promptfoo eval -o results.json. Each test, prompt and provider combination is one case.

A test without a description gets the key test <8 hex characters>, derived from its vars. Changing any var then changes the key. Give every test a description.

With two prompts and two providers, a key looks like Refund above limit | prompts/support_v2.txt | openai:gpt-4o-mini.

evalship JSON

Use evalship JSON for hand-rolled evals or frameworks without JUnit output. The schema is at evalship.com/schema/v1.json.

{
  "schema": "evalship/v1",
  "suite": "support-bot",
  "cases": [
    {
      "key": "refund-above-limit",
      "name": "Declines a refund above 100 euros",
      "passed": true,
      "score": 0.92,
      "threshold": 0.7,
      "duration_ms": 1840,
      "cost_usd": 0.0021,
      "tokens": 812,
      "input": "Can I get a refund of 500 euros?",
      "output": "I can refund up to 100 euros. For larger amounts I'll pass you to a colleague.",
      "file": "evals/support_bot.eval.ts",
      "targets": ["app/support_bot.py#answer", "prompts/support_system.md"],
      "metadata": { "model": "gpt-4o-mini" }
    }
  ]
}

The top-level suite applies to cases without their own. Only cases is required at the top level.

Non-string values in input, output and failure are serialized as JSON before truncation.

Stable keys

evalship matches cases across runs by suite and key. When either changes, the old case shows up as removed and the new one as new, and its flakiness history starts over.

• Keep timestamps, random ids, dates and changing model names out of test names and keys.
• In pytest, give parametrized tests explicit ids= so parameter values don't end up in the key.
• In JUnit, the key includes the module path, so moving or renaming a test file changes its keys.
• In promptfoo, give every test a description.
• If the same suite and key appear twice in one file, the repeats become key (2), key (3) in file order. That depends on order, so give each case its own key.

Up to 20,000 cases per file are read. See Connecting CI for artifact and file size limits.