LangSmith evaluations on every pull request

LangSmith keeps your datasets, your traces and the comparison between experiments. Running its evaluations on every pull request adds a question it leaves to you: which examples did this change break, compared with your default branch, and which would have failed anyway?

What LangSmith's CI setup checks

LangSmith runs evaluations as tests: its pytest plugin and its Vitest and Jest integration "track assertions in LangSmith and raise assertion errors locally (e.g. in CI pipelines)". Each run becomes an experiment in LangSmith. On a pull request, the gate is the test runner's exit code, which goes red when any test fails.

LangSmith's example CI pipeline comments on the pull request with each feedback key's average score next to a fixed criterion, such as >=0.75. A missed criterion doesn't fail the job. Comparing a run with an earlier experiment is a view in LangSmith, not part of the pull request.

Neither tells chance from a break. In the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?). At that rate, a suite of 50 evals goes red on 72% of pull requests that change nothing (the table), and an average can stay above its criterion while one example breaks. Telling the two apart takes each example's history over many runs of your default branch.

How you get results out depends on how your evaluations run.

pytest tests

Tests marked @pytest.mark.langsmith stay ordinary pytest tests, with the same names on every run, so pytest's JUnit XML works as it is. Feedback you log to LangSmith doesn't reach the XML, so record the score next to it:

import pytest
from langsmith import testing as t
from app.support_bot import answer

@pytest.mark.langsmith
def test_refund_limit(record_property):
    question = "Can I get a refund of $500?"
    t.log_inputs({"question": question})
    reply = answer(question)
    t.log_outputs({"answer": reply})
    score = judge_refund_policy(reply)
    t.log_feedback(key="refund_policy", score=score)
    record_property("score", score)
    record_property("threshold", 0.7)
    assert score >= 0.7

LLM evals with pytest has the workflow, and why to set junit_family = "xunit1".

Vitest and Jest tests

ls.test adds a random id to every test name, a new one on each run, as in approves a refundable invoice, test_id=3f9762babea04, and with repetitions it adds run 0, run 1 before it. evalship drops both, so each test keeps one name and its history, and a test's repetitions count as repeats of one eval, which passes when at least half of them pass.

Write JUnit XML as in LLM evals with Vitest or LLM evals with Jest. In Vitest, ls.test doesn't hand your function the test context, so it can't annotate a score, and these evals reach evalship as pass or fail.

evaluate() from a script

With client.evaluate(), write each example's result to a file that CI keeps, next to reporting to LangSmith. This needs langsmith 0.7.22 or later, for results.url:

import json
from langsmith import Client
from app.support_bot import answer

THRESHOLD = 0.8
client = Client()

def target(inputs: dict) -> dict:
    return {"answer": answer(inputs["question"])}

def exact(outputs: dict, reference_outputs: dict) -> bool:
    return outputs["answer"] == reference_outputs["answer"]

results = client.evaluate(target, data="support-bot", evaluators=[exact], experiment_prefix="ci")

evals = []
for row in results:
    run, example = row["run"], row["example"]
    scores = [float(e.score) for e in row["evaluation_results"]["results"] if e.score is not None]
    score = min(scores) if scores else None
    evals.append({"key": str(example.id), "score": score, "threshold": THRESHOLD,
                  "passed": run.error is None and score is not None and score >= THRESHOLD,
                  "failure": run.error, "input": example.inputs, "output": run.outputs,
                  "url": run.get_url()})

json.dump({"schema": "evalship/v1", "suite": "support-bot", "url": results.url, "evals": evals},
          open("eval-results.json", "w"), default=str)

name: evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python -m evals.run_evaluation # -m, so the script can import app/ from the repository root
        env:
          LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-results
          path: eval-results.json

The lowest score, not the average. With several evaluators, one that fails fails the example. Boolean evaluators score True or False, hence float(). Keep scores between 0 and 1.
Keys that last. Each example is known by its id, which stays the same while you edit the dataset and changes when you recreate it. Recreating the dataset starts every example's history over.
Links. run.get_url() asks LangSmith once per example. In TypeScript, the results have no URL, so leave url out.

How evalship does this

evalship reads the results file your workflow uploads and compares every pull request with your default branch, example by example. It sets aside examples that couldn't run (a missing key, a rate limit, a retired model), examples that are flaky on main, and failures within the noise it measures over each suite's last 30 runs. Then it comments on the pull request with what's left, and with evaluate(), each changed example links to its run in LangSmith. How the PR comment works has the details.

With this in .evalship.yml, its check fails a pull request that breaks an example that passed on main:

gate:
  fail_on_new_failures: true

evalship never sees your LangSmith key: your evaluations run in your own GitHub Actions, and evalship reads only the file they upload. It's free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart). The agent writes the workflow for your repository.