Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

Articles · November 17, 2026

Mastra evals on every pull request

Mastra ships its own evals: scorers, quick checks, gates and a Vitest integration. Its guide to running them in CI ends at a test that passes or fails. On a pull request that isn't the question. The question is whether the change broke an eval that passed on main.

What Mastra's evals report in CI

runEvals runs your agent over a list of items and returns each scorer's average. Gates and thresholds turn that into a verdict: passed, scored or failed. With @mastra/evals/vitest, expectEvals(...).toPass() fails the Vitest test when the verdict does.

The verdict comes from this run alone. A threshold of 0.7 says nothing about what the score was on main, and LLM evals fail by chance. In the CI of five public repositories, a passing eval failed on the next run about 1 time in 40: 2.5% across the five, from 0% to 5.3% per repository (How noisy are LLM evals in CI?). A red run can be the change, or an eval that flips on every tenth run. Telling them apart takes each eval's history on your default branch.

Mastra's Vitest reporter prints the scores to the console. None of them reach the JUnit report, so CI keeps pass and fail only.

One test per item, not one per dataset

expectEvals takes the whole dataset in one test. In the JUnit report that is one result, whatever the number of items. With toPass(0.8), an item that broke hides in a pass rate that still clears the bar.

expectEval takes one item. With test.for, each item is its own test:

import { test } from "vitest"
import { expectEval } from "@mastra/evals/vitest"
import { checks } from "@mastra/evals/checks"
import { mastra } from "../support-agent"

const cases = [
  { id: "double-charge", input: "I was charged twice for my subscription this month.", groundTruth: "billing" },
  { id: "app-crash", input: "The app crashes every time I open the settings page.", groundTruth: "bug" },
]

test.for(cases)("$id", { timeout: 60_000 }, async ({ input, groundTruth }, { annotate }) => {
  await expectEval({
    target: mastra.getAgent("supportAgent"),
    data: { input, groundTruth },
    // checks.equals ignores case but not whitespace around the reply
    gates: [checks.matches(new RegExp(`^\\s*${groundTruth}\\s*$`, "i"))],
    onItemComplete: ({ targetResult }) => annotate(targetResult.text, "output"),
  }).toPass()
})

A broken item is then one failing test with its own history, and the others keep theirs.

Name each test by an id. Mastra's own example names them "weather agent: $input", so editing an item's input renames its test, and the eval starts over with no history.

Write JUnit XML, with the answer and the score

Vitest's built-in reporter writes the report. The workflow runs on pull requests and on every push to main, so each pull request has results to compare with:

name: evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      - run: npm ci
      - run: npx vitest run evals --reporter=default --reporter=junit --outputFile.junit=eval-results.xml
        continue-on-error: true # the comparison with main decides, not one failure
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

A pass keeps no score, and a failure doesn't say what the agent answered. onItemComplete gets both, and Vitest's annotate writes them to the report as properties of the test:

await expectEval({
  target: mastra.getAgent("supportAgent"),
  data: { input, groundTruth },
  scorers: [{ scorer: checks.similarity(groundTruth), threshold: 0.7 }],
  onItemComplete: async ({ targetResult, scorerResults }) => {
    await annotate(targetResult.text, "output")
    await annotate(String(scorerResults["check-similarity"].score), "score")
    await annotate("0.7", "threshold")
  },
}).toPass()

The XML then carries <property name="score" value="0.41"/> on the test, for a pass and for a failure. A score that drops from 0.95 to 0.72 against a threshold of 0.7 still passes, and that's the drift worth seeing before it fails.

Leave out retry. In the JUnit report, a test that failed once and passed on its second try is a plain pass, so a flaky eval looks as solid as one that never fails.

Agents defined in one file and called from another

A Mastra agent's instructions, model and tools sit in its new Agent({...}). The code that calls it is often elsewhere, or outside the repository when Mastra's server runs the agent. evalship reads the definition as the AI call, so a pull request that changes an agent's instructions, model or tools without an eval for it gets a note. Say which agent an eval exercises with targets:

await annotate("src/mastra/agents/support-agent.ts", "targets")

Untested changes has the other ways an eval counts for a change.

The same agent in production

Mastra's OpenTelemetry exporter sends each run's model calls and tool calls. Pointed at evalship, Monitor checks them against what your evals test, and each model call takes its agent's name. Already tracing with Mastra has the config.

How evalship does this

evalship reads the eval-results.xml your workflow uploads and compares every pull request with your default branch, eval by eval. It sets aside evals that couldn't run (a missing key, a rate limit, a retired model), evals that are flaky on main, and failures within the noise it measures over each suite's last 30 runs. It reads the score and threshold you annotate, so a drop shows before a gate fails. Then it comments on the pull request with what's left. How the PR comment works has the details.

With this in .evalship.yml, its check fails a pull request that breaks an eval that passed on main:

gate:
  fail_on_new_failures: true

evalship is free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart). The agent writes the workflow above for your repository.

Tested with @mastra/core 1.83, @mastra/evals 1.10 and Vitest 5.1.

Further reading