Evals, eval harness, agent harness, harness engineering: what each term means
Four terms share two words, and articles use them for different things. An eval is a test. An eval harness runs evals. An agent harness is what the evals test. Harness engineering is the work of improving that agent harness, with evals as one of its checks. None of them names what happens after the harness has run, so this article ends with a taxonomy that does. Definitions and sources as of November 18, 2026.
| Term | What it is | Example |
|---|---|---|
| Eval | A test of an AI call or an agent: an input, a way to grade the output, a pass or fail | "Refund request" must cite the 30-day policy |
| Eval suite | The test cases for one behavior or one prompt | 20 support questions for the refund prompt |
| Eval harness | The code that runs evals: it loads the test cases, calls your code, grades, repeats and reports | promptfoo, Inspect, pytest with a judge, tinyeval |
| Agent harness | Everything around the model that makes it an agent: prompts, tools, the loop, memory | Claude Code, Codex, your own agent loop |
| Harness engineering | Changing the agent harness so the agent stops making a mistake | A rule in AGENTS.md, a linter, an eval |
| Test harness | The older software term: the code that runs tests against a unit, with its drivers and stubs | JUnit, pytest |
An eval is a test. An eval harness runs it
Anthropic's glossary from January 9, 2026 is the one most articles borrow (Demystifying evals for AI agents). A task is "a single test with defined inputs and success criteria". A grader is "logic that scores some aspect of the agent's performance". And: "An evaluation harness is the infrastructure that runs evals end-to-end."
So the difference is content and machinery. The evals are yours: the inputs you chose and what a good answer is. The harness is plumbing, and it does five jobs:
- Loads the test cases.
- Calls your real code with each one.
- Grades each answer, in code or with an LLM judge.
- Repeats, because the same input can pass once and fail once.
- Writes the results where CI and people can read them.
You can swap the harness and keep the evals. A team that says "we have evals" and a team that says "we built an eval harness" may have the same thing. The second team is telling you they wrote the plumbing themselves.
Eval tests is another name for evals. It stresses that they run like tests: in a test runner, on every change.
The names for the parts differ by tool. Anthropic's "task" is a "test case" in promptfoo and Braintrust and a "dataset item" in Langfuse.
An older meaning is still around. EleutherAI's lm-evaluation-harness is "a framework for few-shot evaluation of language models". It runs public benchmarks to compare models. An eval harness in a 2026 article usually tests your own product on your own test cases.
The agent harness is what the evals test
"Agent = Model + Harness" comes from LangChain's Vivek Trivedy on March 10, 2026: "A harness is every piece of code, configuration, and execution logic that isn't the model itself" (The anatomy of an agent harness). Anthropic calls it "the system that enables a model to act as an agent".
The two harnesses meet in one sentence from Anthropic: "When we evaluate 'an agent,' we're evaluating the harness and the model working together." Change the system prompt, a tool description or the model, and you changed the thing under test.
Articles disagree on which harness contains the other. DeepEval writes that "the eval harness is part of, not a replacement for, the broader agent harness" (What is an eval harness). Anthropic has the eval harness run the agent from outside. Both happen:
- When you test an agent, the eval harness wraps it. It starts the agent, records each step and grades the outcome.
- When a coding agent checks its own work, the eval harness is one of its tools. The agent edits a prompt, runs the evals and reads the result.
Harness engineering is the practice. Evals are one of its checks
Mitchell Hashimoto named it on February 5, 2026. Each time an agent makes a mistake, "you take the time to engineer a solution such that the agent never makes that mistake again" (My AI adoption journey). His two forms are a line in AGENTS.md and a tool the agent can run to check its work.
Six days later, OpenAI's Ryan Lopopolo described a product built with no hand-written code. The engineers' job became to "design environments, specify intent, and build feedback loops" (Harness engineering). The post never uses the word "evals". "Evaluation harnesses" appears once, in a list of things the agents produce.
Birgitta Böckeler's article on martinfowler.com splits the harness in two (Harness engineering for coding agent users, April 2, 2026):
- Guides "anticipate the agent's behaviour and aim to steer it before it acts". An AGENTS.md rule is a guide.
- Sensors "observe after the agent acts and help it self-correct". Tests, linters and LLM judges are sensors.
Evals are sensors. Google's Taylor Mullen and Christian Gunderman say so directly: "behavioral evaluations are a core pillar of harness engineering" (The anatomy of harness engineering, September 9, 2026).
Which one you need
- Your product makes AI calls: a prompt that writes a summary, a model that sorts tickets. You need evals and a small harness to run them. Harness engineering isn't your problem.
- You build an agent. The agent harness is your product. Your evals grade what the agent did and where it ended up, and each harness change is a change to test.
- You write code with a coding agent. You're doing harness engineering already, each time you add a rule to AGENTS.md. Evals belong in that harness: tell the agent to run them after it changes a prompt.
What a harness doesn't tell you
A harness runs the evals and counts passes. Three questions are left.
- Can the evals fail? Böckeler asks it of every sensor: "If sensors never fire, is that a sign of high quality or inadequate detection mechanisms?" In 10 open-source eval suites we read, every one had a test case that couldn't fail (LLM evals that can't fail).
- Is a failure real? In CI on five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?).
- Did this change break it? Most harnesses run the pull request's code only, with no comparison to main (LLM evals on pull requests).
A taxonomy that goes past the harness
The glossaries above name what runs and what is run. They stop at the score. The three questions have no shared words yet, and neither does production. Here is the taxonomy we use at evalship, one question per layer.
| Question | Term | Meaning | Elsewhere |
|---|---|---|---|
| What is tested? | AI call | One call to a model in your code: a prompt, a model and its settings | LLM call |
| Agent | A model and its agent harness, making AI calls in a loop | ||
| What tests it? | Test case | One input | Task (Anthropic), dataset item (Langfuse) |
| Eval | One pass or fail on a test case: one thing checked about the answer | Grader or assertion (Anthropic), score (Braintrust, Langfuse) | |
| Suite | The test cases for one behavior | Evaluation suite (Anthropic) | |
| What runs it? | Eval harness | The code that runs a suite | |
| Run | One pass of the suites at one commit | ||
| Can the tests be trusted? | Flaw | A defect in the evals: a test case that can't fail, a judge that grades its own model | No common name |
| Untested AI call | An AI call or prompt that no eval runs | Coverage gap | |
| What did this change do? | Regression | An eval that passed on main and fails on the change, beyond chance | |
| Flaky eval | An eval that passes and fails on the same code | ||
| Too early to tell | An eval that got worse in a suite main has run too few times to judge | ||
| What happens in production? | Issue | A production AI call that breaks a rule your evals test | |
| What would be better? | Experiment | One change to an AI call, run on the same evals as the current version |
Two choices in it are deliberate.
A test case and an eval are different things. Most writing calls both "an eval". The counts differ: one test case with five things checked is five evals. "We have 40 evals" can mean 40 inputs or 8.
The AI call is the unit. A flaw, a regression, an issue and an experiment each belong to one AI call. That lets one page answer, per AI call: is it tested, did the last change break it, and does it hold up in production.
Where evalship fits
evalship doesn't replace your eval harness. It covers the layers after it, and sets up the harness when you have none.
| Layer | What evalship does |
|---|---|
| What runs it? | Reads the results your harness writes in CI or on your machine (formats). With no harness yet, your coding agent writes one during setup, or uses tinyeval: one bash file, any language |
| Can the tests be trusted? | Setup reviews your evals and reports each flaw with the file and line (the checks it runs). Every pull request names the untested AI calls it changes (untested changes) |
| What did this change do? | The pull request comment sets regressions apart from flaky evals and from chance (how the PR comment works) |
| What happens in production? | Monitor checks each production AI call against the rules your evals test (Monitor) |
| What would be better? | Optimize has your coding agent run experiments on your evals, and says which win beyond chance (Optimize) |
It also adds one sensor to your coding agent's harness. Setup puts a line in AGENTS.md: before the agent finishes a change to a prompt or a model, it runs the evals the change affects (what your agent does).
Setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart).