Skip to content

Articles · October 30, 2026

Langfuse experiments on every pull request

Langfuse keeps your datasets, your traces and the comparison between experiments. Running an experiment on every pull request adds a question Langfuse leaves to you: which test cases did this change break, compared with your default branch, and which would have failed anyway?

What Langfuse's experiment action checks

Langfuse's CI/CD guide runs your experiment script with langfuse/experiment-action. The action comments on the pull request with each script's status, its run-level scores such as avg_accuracy, a table of up to 50 test cases with their outputs and scores, and a link to the experiment in Langfuse. It fails the job when the script raises RegressionError, which your code raises when a score misses the threshold you set.

The pass or fail is the threshold you write, and the guide is frank about its limits: "An aggregate threshold can miss a newly failing case." For a gate per test case, it says to keep a reviewed baseline file in your repository, with a pass or fail verdict for every case, and to update it by hand through code review: "Never generate the approved artifact automatically from the candidate being tested."

A hand-kept baseline runs into LLM noise. In the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?). Against a fixed list of verdicts, a dataset of 50 cases misses at least one on 72% of runs that change nothing (the table), and the file can't say which cases flip on main and which never do. That takes each case's history over many runs of your default branch.

Run the experiment from a script and write the results

Run the experiment as a plain script on pull requests and on pushes to your default branch. Next to reporting to Langfuse, it writes each test case's result to a file that CI keeps. This is for the Python SDK v5; the notes below cover earlier versions.

import json
from langfuse import get_client, Evaluation
from app.support_bot import answer

THRESHOLD = 0.8
langfuse = get_client()

def task(*, item, **kwargs):
    return answer(item.input)

def exact(*, output, expected_output, **kwargs):
    return Evaluation(name="exact", value=float(output == expected_output))

def trace_url(trace_id):
    try:
        return langfuse.get_trace_url(trace_id=trace_id)
    except Exception:  # a missing link shouldn't fail the run
        return None

dataset = langfuse.get_dataset("support-bot")
result = dataset.run_experiment(name="ci", task=task, evaluators=[exact])

evals, ran = [], set()
for r in result.item_results:
    ran.add(r.item.id)
    scores = [e.value for e in r.evaluations if isinstance(e.value, (int, float))]
    score = min(scores) if scores else None
    evals.append({"key": r.item.id, "score": score, "threshold": THRESHOLD,
                  "passed": score is not None and score >= THRESHOLD,
                  "input": r.item.input, "output": r.output, "url": trace_url(r.trace_id)})
# run_experiment leaves out items whose task raised, so add them back as failures
evals += [{"key": item.id, "passed": False, "input": item.input, "failure": "The task raised an error."}
          for item in dataset.items if item.id not in ran]

json.dump({"schema": "evalship/v1", "suite": "support-bot", "url": result.experiment_url, "evals": evals},
          open("eval-results.json", "w"), default=str)

name: evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python -m evals.run_experiment # -m, so the script can import app/ from the repository root
        env:
          LANGFUSE_PUBLIC_KEY: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
          LANGFUSE_SECRET_KEY: ${{ secrets.LANGFUSE_SECRET_KEY }}
          LANGFUSE_BASE_URL: https://cloud.langfuse.com
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-results
          path: eval-results.json

Every run is still an experiment in Langfuse, so its comparison view keeps working.

To keep the experiment action and its comment, run the same loop inside your experiment(context) function, on the result context.run_experiment returns, and write the file to os.path.join(os.environ["GITHUB_WORKSPACE"], "eval-results.json"). Upload it in a step after the action, with if: always(), since the action fails the job on a RegressionError.

What the script takes care of

  • Test cases whose task raised. run_experiment leaves them out of item_results. Without the last loop, a case that crashes vanishes from the results, and the run looks better for it.
  • The lowest score, not the average. With several evaluators, one that fails fails the test case. An average can hide a hard rule that broke. Keep scores between 0 and 1.
  • Keys that last. Each test case is known by its dataset item id, which stays the same while you edit the dataset and changes when you recreate it. Recreating the dataset starts every case's history over.
  • SDK versions. run_experiment arrived in Python SDK 3.4.0. Before v5, the experiment's link is result.dataset_run_url. Before 3.8.1, the SDK reads LANGFUSE_HOST rather than LANGFUSE_BASE_URL. In TypeScript, the fields are camelCase: result.itemResults, r.traceId, result.experimentUrl and await langfuse.getTraceUrl(traceId).

How evalship does this

evalship reads the eval-results.json your workflow uploads and compares every pull request with your default branch, test case by test case. Its baseline is your default branch's own runs, so there's no file to keep by hand. It sets aside cases that couldn't run (a missing key, a rate limit, a retired model), cases that are flaky on main, and failures within the noise it measures over each suite's last 30 runs. Then it comments on the pull request with what's left, and each changed case links to its trace in Langfuse. How the PR comment works has the details.

With this in .evalship.yml, its check fails a pull request that breaks a case that passed on main:

gate:
  fail_on_new_failures: true

evalship never sees your Langfuse keys: the script runs in your own GitHub Actions, and evalship reads only the file it uploads. It's free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart). The agent writes the script and the workflow for your repository.

Further reading