A Langfuse alternative with less to set up
Langfuse is an open-source platform for AI applications: tracing, prompt management, datasets, experiments, evaluators, dashboards and an AI gateway. evalship does a smaller job. It says which evals a pull request broke, checks every AI call in production, and tells you about the issues it finds. It's an alternative when that job is why you looked at Langfuse, and it runs next to Langfuse when you need the rest. This comparison is from both tools' docs and pricing pages as of November 20, 2026.
Langfuse and evalship side by side
| Langfuse | evalship | |
|---|---|---|
| Where evals live | Datasets in Langfuse. Experiments run through its SDK, its UI or OpenTelemetry | Test files in your repository, run by your own command: pytest, Vitest, Jest, promptfoo or tinyeval. evalship reads the results file |
| On a pull request | Its GitHub Action comments with each script's scores and up to 50 items. Your script sets the threshold that fails the job. No comparison with main | Compares each eval with its result on your default branch, sets aside flaky evals and failures within chance, and comments with what's left. Its check can block the merge |
| The evals themselves | Calibrates an LLM judge against human labels | Your coding agent reviews every suite at setup and lists its flaws, such as a test case that can't fail |
| In production | Traces, sessions, users, tokens and cost. You add LLM-as-a-judge evaluators, dashboards and alerts | Checks every AI call with no evaluator to write: failed calls, overpromises, made-up facts, private data, frustrated users. Each issue comes by email, with a line for your coding agent to fix it |
| Setup | Instrument your app with its SDK or OpenTelemetry, then build datasets and evaluators | One line to your coding agent, which writes or repairs the evals and the CI workflow |
| Prompt management, playground, annotation queues, custom dashboards, AI gateway | Yes | No. Prompts stay in your code, where a pull request shows every change |
| Open source and self-hosting | Yes | Not yet |
| Price | Hobby: free, 50k units a month, 2 users, 30 days of data. Core: $29 a month, 100k units, unlimited users, 90 days. Pro: $199 a month | Free: 2 users, 1 private repository, 10,000 production calls a month, 30 days of eval history. Pro: $29 a month, unlimited users and repositories, 500,000 production calls, 1 year. Business: $99 a month |
A unit in Langfuse is a trace, an observation (a step within a trace) or a score. evalship is free for public repositories on every plan.
What you set up
For evals on pull requests, Langfuse's CI/CD guide has you store a dataset in Langfuse, write an experiment script with its SDK, add langfuse/experiment-action with your keys, and choose what fails the job: a score threshold, or a baseline file you review by hand. For production, you instrument the app, then set up the evaluators that score live traces, and your dashboards and alerts.
With evalship you paste one line into your coding agent: Set up https://evalship.com/setup.md. The agent finds your AI calls and the evals that test them, reviews the evals, and writes the CI workflow. With no evals yet, it offers to write a first small suite (quickstart). Setup then goes on with production: the agent adds one call after each AI call, or you point OpenTelemetry at evalship (Monitor).
When evalship fits better
- Your evals belong with your code. They are test files, reviewed in the same pull request as the prompt they test, in any language whose test runner writes JUnit XML (the formats evalship reads).
- You want the pull request to say what broke. A fixed threshold goes red by chance: in the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?). evalship compares each eval with its history on main, so a flaky one doesn't block a merge and a real break does.
- You don't know whether your evals would catch a break. In 10 of 10 open-source eval suites, at least one test case couldn't fail (the flaws we found). evalship's review looks for those before you rely on the suite.
- You want to hear about production problems before you build evaluators. The checks on answers run from the first call.
When to stay with Langfuse
- People outside the codebase edit prompts. Langfuse versions prompts, releases them by label and has a playground. evalship has none of that.
- Your team labels outputs by hand. Langfuse has annotation queues. evalship has no human review step.
- Data can't leave your network. Langfuse can be self-hosted. evalship is hosted in the EU and can't be self-hosted yet (security).
- You want your own dashboards for cost and latency, or one gateway for every model provider.
- Your security review asks for a SOC 2 or ISO 27001 report. Langfuse's Pro plan includes both. evalship is working towards SOC 2.
Using both
Nothing has to move. Your experiment script keeps reporting to Langfuse and also writes a results file, about 15 lines more. evalship then compares every pull request with main, and each changed eval links to its trace in Langfuse (Langfuse experiments on every pull request). In production, an app that exports OpenTelemetry traces can add evalship as a second exporter (Monitor).
Further reading
- LLM evals on pull requests: how the tools compare: Langfuse, Braintrust, promptfoo, LangSmith, DeepEval and Raindrop on a pull request.
- A Braintrust alternative that keeps evals in your repo: the same comparison for Braintrust.
- Langfuse, Pricing and Evaluation overview: the plans and features compared here.
- Pricing and billing: evalship's plans and limits.