Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

Articles

Thought leadership

AI agent monitoring: what to watch in production

What to watch once an AI agent is in production: failed tool calls, retry loops and answers its tools contradict, and what a trace needs to catch them. November 20, 2026

Evals, eval harness, agent harness, harness engineering: what each term means

An eval is a test, an eval harness runs it, and harness engineering improves a different harness. The terms sorted, and a taxonomy for what comes after. November 18, 2026

Findings

Recorded LLM replies in Rails tests pass any prompt change

In 16 of 19 Ruby apps that replay their AI calls in tests, the replay ignores the request, so any prompt or model change passes. November 6, 2026

LLM evals that can't fail: flaws in 10 open-source eval suites

Every one of 10 open-source LLM eval suites had a test case that couldn't fail. The flaws we found most, with examples and how to check yours. November 2, 2026

LLM evals on pull requests: how the tools compare (2026)

promptfoo, Braintrust, Langfuse, LangSmith, DeepEval and Raindrop on pull requests: what each compares with main, and what fails the check. November 2, 2026

LLM as a judge: five mistakes in real eval suites

In 6 of 9 open-source eval suites, the LLM judge had a flaw, most often grading its own model's answers. Five mistakes, and how to fix each. November 2, 2026

How noisy are LLM evals in CI?

How often passing LLM evals fail by chance in CI, measured on public repositories, and how big a break your suite can see through it. October 8, 2026

Guides by tool

A Langfuse alternative with less to set up

evalship next to Langfuse, from both tools' docs and prices: what each does on pull requests and in production, when to switch and when to stay. November 20, 2026

A Braintrust alternative that keeps evals in your repo

evalship next to Braintrust, from both tools' docs and prices: evals in your repo, a pull request check that ignores chance, and when to stay. November 20, 2026

Mastra evals on every pull request

Mastra's evals run in Vitest. Get one result per dataset item into CI, keep answers and scores in the JUnit report, and compare each pull request with main. November 17, 2026

LLM evals in Ruby, Go and Rust, without Python or Node

Run LLM evals in a Ruby, Go or Rust app without adding Node or Python to CI: call your real code, check in code first, run each test case 3 times. November 5, 2026

Run your Claude evals in CI on your Claude plan

Claude Max, Ultra and Team plans include $100 to $500 a month of API credit. It pays for evals in CI. Subscriptions cover only Claude Code steps. November 1, 2026

Langfuse experiments on every pull request

What Langfuse's experiment action checks, a script that keeps each test case's result in CI, and why a hand-kept baseline file runs into noise. October 30, 2026

Braintrust evals on every pull request

What Braintrust's eval action reports and what fails it, with a script and workflow that keep each test case's result to compare with main. October 30, 2026

LangSmith evaluations on every pull request

Get a result per example out of LangSmith in CI, from pytest, Vitest, Jest or evaluate(), and see what its example pipeline doesn't check. October 30, 2026

LLM evals with pytest in GitHub Actions

Run LLM evals with pytest in GitHub Actions: scores in the JUnit XML, test names that last, and a build that's red only when something broke. October 30, 2026

LLM evals with Vitest in GitHub Actions

Run LLM evals with Vitest in GitHub Actions: JUnit XML with the console kept, scores recorded with annotate, and why retry hides flaky evals. October 30, 2026

LLM evals with Jest in GitHub Actions

Run LLM evals with Jest in GitHub Actions: jest-junit settings that keep names stable, scores in the XML, and why retries hide flaky evals. October 30, 2026

Replacing OpenAI Evals: promptfoo in CI without flaky reds

OpenAI Evals' shutdown date is November 30, 2026. Run the evals in promptfoo on every pull request, without a gate that goes red when nothing broke. October 26, 2026