Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

Case study · PrefectHQ/prefect-mcp-server

14 failing evals that never got graded

Prefect's MCP server gives AI assistants read-only tools for inspecting Prefect workspaces. Its evals read like support cases: set up a broken workspace, ask the agent what a user would ask, and have a model check that the answer names the real cause. In August 2026, pull request #200 fixed two tools, and its eval report came back 14 failing. None of those evals had been graded: the model that grades them had been retired.

14 of 20
evals the repo's eval report listed as failing on the first push
0 of 14
of those were graded: the judge model returned 404
0
regressions evalship reported; it set the 14 aside as couldn't run, with the error
5
flaws evalship's review found in its evals

What evalship's review found

At setup, your coding agent reads every scorer, judge and prompt, and writes down where the evals fall short. On PrefectHQ/prefect-mcp-server at 2940a3e, the commit #200 branched from, it found:

1 Eval gapUntested prompt

9 prompts without an eval

A change that breaks one of them reaches your users without a failing eval.

Found in 9 places
  • src/prefect_mcp_server/server.py:80 get_identity tool description
  • src/prefect_mcp_server/server.py:295 get_task_runs tool description
  • src/prefect_mcp_server/server.py:387 read_events tool description
  • src/prefect_mcp_server/execution_plans.py:187 execution_plans_validate tool description
  • src/prefect_mcp_server/execution_plans.py:212 execution_plans_get_schema tool description
  • src/prefect_mcp_server/execution_plans.py:262 execution_plans_get tool description
  • src/prefect_mcp_server/execution_plans.py:319 execution_plans_publish tool description
  • packages/docs_mcp_server/docs_mcp_server/_server.py:86 search_prefect tool description
  • plugins/prefect/skills/workflows/SKILL.md:1 workflows skill (hosted plugin instructions)
Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: 9 prompts in our product code have no eval:
- src/prefect_mcp_server/server.py:80 (get_identity tool description)
- src/prefect_mcp_server/server.py:295 (get_task_runs tool description)
- src/prefect_mcp_server/server.py:387 (read_events tool description)
- src/prefect_mcp_server/execution_plans.py:187 (execution_plans_validate tool description)
- src/prefect_mcp_server/execution_plans.py:212 (execution_plans_get_schema tool description)
- src/prefect_mcp_server/execution_plans.py:262 (execution_plans_get tool description)
- src/prefect_mcp_server/execution_plans.py:319 (execution_plans_publish tool description)
- packages/docs_mcp_server/docs_mcp_server/_server.py:86 (search_prefect tool description)
- plugins/prefect/skills/workflows/SKILL.md:1 (workflows skill (hosted plugin instructions))
For each, add cases in a new eval suite that runs on pull requests: inputs that reach the prompt the way production calls it, and checks on what its output must and must not say. Name each eval after the prompt it covers (its function, constant or file), so it's clear which prompt it tests. If one isn't worth an eval, say which and why instead of adding a weak one.
Fix it in a pull request; evalship confirms the fix on it.
2 ImportantEval bug

Three concurrency evals blame a limit that no run is using

In the work-pool, work-queue and tag cases every run is forced to Late and nothing occupies the one slot. So the judge passes an agent that blames any configured limit of 1, and fails one that checks usage and correctly says the limit is free. For the tag case, the tool output itself says over_limit: false. No case has a limit that is set but free, so a PR that makes the agent, or the queue hint, blame idle limits stays green.

evals/late_runs/test_work_pool_concurrency.py:63
# Create multiple flow runs - first will consume the slot, others will be Late

⚠ lines 73-76 then force all three runs to Late, so nothing holds the slot

evals/late_runs/test_work_pool_concurrency.py:132
that its concurrency limit is exhausted/full, not just give generic

⚠ the judge requires 'exhausted' for a limit with no run in it

evals/late_runs/test_work_queue_concurrency.py:131
pool/queue name and that the queue's concurrency limit is exhausted,

⚠ this fixture also forces every run to Late (line 82)

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: Three concurrency evals blame a limit that no run is using. In the work-pool, work-queue and tag cases every run is forced to Late and nothing occupies the one slot. So the judge passes an agent that blames any configured limit of 1, and fails one that checks usage and correctly says the limit is free. For the tag case, the tool output itself says over_limit: false. No case has a limit that is set but free, so a PR that makes the agent, or the queue hint, blame idle limits stays green. See evals/late_runs/test_work_pool_concurrency.py:63, evals/late_runs/test_work_pool_concurrency.py:132, and evals/late_runs/test_work_queue_concurrency.py:131. Fill the slot the way test_deployment_concurrency.py:71-77 does, with a Running flow run in the pool or queue, and rethink the tag case, since its limit counts tasks a Late run never starts. Add one case with a free limit of 1, where blaming the limit fails.
Fix it in a pull request; evalship confirms the fix on it.
3 ImportantEval bug

The eval shell tool reports every successful prefect command as exit code 124

All four evals that change state through the Prefect CLI (trigger a run, cancel late runs, create a reactive or proactive automation) show the agent a timeout exit code for every command that worked. An agent that sensibly retries fails the trigger case's single-run assert, and one that ignores exit codes passes. Real clients run the CLI in a real shell, so these cases measure how the model copes with a broken tool, not how it uses the CLI.

evals/_tools/run_shell_command.py:49
exit_code=proc.returncode or 124, stdout=out.decode(), stderr=err.decode()

⚠ a successful command returns 0, which is falsy, so the agent sees 124

evals/_tools/run_shell_command.py:47
return ShellResult(exit_code=124, stdout="", stderr="Timed out after 20s")

⚠ 124 is the tool's own timeout code

evals/test_trigger_deployment_run.py:75
assert len(flow_runs) == 1

⚠ an agent that retries the apparently failed `prefect deployment run` creates a second run and fails here

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: The eval shell tool reports every successful prefect command as exit code 124. All four evals that change state through the Prefect CLI (trigger a run, cancel late runs, create a reactive or proactive automation) show the agent a timeout exit code for every command that worked. An agent that sensibly retries fails the trigger case's single-run assert, and one that ignores exit codes passes. Real clients run the CLI in a real shell, so these cases measure how the model copes with a broken tool, not how it uses the CLI. See evals/_tools/run_shell_command.py:49, evals/_tools/run_shell_command.py:47, and evals/test_trigger_deployment_run.py:75. Return proc.returncode unchanged (it is set once communicate() returns) and keep 124 for the timeout branch only. Add a unit test next to tests/test_eval_tool_spy.py that runs `prefect version` through the tool and expects exit code 0.
Fix it in a pull request; evalship confirms the fix on it.
4 ImportantSuspiciously easy suite

The automation-not-firing case puts its answer in the automation's description

The agent can answer 'it needs 3 failures and only 1 happened' from the compact get_automations listing without reading the trigger. A PR that breaks the detail path (the id lookup at automations.py:22-29 or the trigger dump at line 74) still passes this case, which is the only one about debugging automations.

evals/automations/test_debug_not_firing.py:28
description="Automation that triggers after 3 consecutive failures",

⚠ states the answer; it is also inexact, since threshold=3 within=300 counts any 3 failures in 5 minutes

src/prefect_mcp_server/_prefect_client/automations.py:66
"description": automation.description,

⚠ included in the compact listing; the trigger config appears only in detail mode (line 74)

evals/automations/test_debug_not_firing.py:81
"Does the agent identify that the automation has a threshold of 3 but only 1 failure occurred, "

⚠ the description plus the one failure named in the user's question are enough to pass

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: The automation-not-firing case puts its answer in the automation's description. The agent can answer 'it needs 3 failures and only 1 happened' from the compact get_automations listing without reading the trigger. A PR that breaks the detail path (the id lookup at automations.py:22-29 or the trigger dump at line 74) still passes this case, which is the only one about debugging automations. See evals/automations/test_debug_not_firing.py:28, src/prefect_mcp_server/_prefect_client/automations.py:66, and evals/automations/test_debug_not_firing.py:81. Give the fixture a neutral description and assert that get_automations was called with an id filter for this automation, so the agent has to read trigger.threshold.
Fix it in a pull request; evalship confirms the fix on it.
5 WarningSuspiciously easy suite

The log-correlation rate-limit case passes without checking rate limits

This is the only case that starts from log evidence, and the step it exists to check (reaching for review_rate_limits) is optional. A PR that rewrites the review_rate_limits description (server.py:518-541) so the agent stops using it on logs still passes: the agent reads the 429 lines, says it looks like rate limiting, and offers to check.

evals/rate_limits/test_cloud_correlate_logs.py:81
could offer to review the rate limits as the next step of investigation.

⚠ the judge accepts an answer that never calls review_rate_limits

evals/rate_limits/test_cloud_correlate_logs.py:26
logger.warning("Prefect API returned 429 Too Many Requests")

⚠ the flow's own logs already name the 429

evals/README.md:66
verifies agent can correlate 429 warnings in flow logs with rate limit data (Cloud)

⚠ no rate-limit eval asserts that review_rate_limits ran

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: The log-correlation rate-limit case passes without checking rate limits. This is the only case that starts from log evidence, and the step it exists to check (reaching for review_rate_limits) is optional. A PR that rewrites the review_rate_limits description (server.py:518-541) so the agent stops using it on logs still passes: the agent reads the 429 lines, says it looks like rate limiting, and offers to check. See evals/rate_limits/test_cloud_correlate_logs.py:81, evals/rate_limits/test_cloud_correlate_logs.py:26, and evals/README.md:66. Drop the 'Alternatively' sentence from the rubric, and add tool_call_spy.assert_tool_was_called("review_rate_limits") to this case.
Fix it in a pull request; evalship confirms the fix on it.

What evalship would have said on #200

  1. 14 evals fail, and none of them ran

    The repo's eval report compares counts with the base branch: “14 ❌ +13”. Every one of the 14 failures is the same error, a 404 for claude-opus-4-1-20250805, the model that grades the agent's answers. The agent ran; nothing graded it.

    evalship reads each failure's message, sees an eval that never got a verdict, and says so in one line with the error, instead of counting 14 regressions. The 6 evals that were graded are compared with main as usual. Its reviewer adds what the eval report can't: none of the rate-limit evals calls the tool with the new workspace_id parameter, so the fix itself is only covered by unit tests. That's an eval flaw, so evalship's check fails until someone adds the case or dismisses it.

    github.com/PrefectHQ/prefect-mcp-server/pull/200

    evalship-appbot commented 2 minutes ago

    evalship report

    🛑 Check failed: fix 1 eval flaw, or dismiss it in the report.

    🔌 Couldn't run: 14 evals in debug_not_firing, latest_release_notes, deployment_concurrency, and 11 more failed before they were scored (pydantic_ai.exceptions.ModelHTTPError: status_code: 404, model_name: claude-opus-4-1-20250805, body: {'type': 'error'...), so they aren't compared with main@2940a3e.

    ⚠️ Eval gap: The new workspace_id parameter on review_rate_limits is only covered by unit tests; the rate-limits-and-cloud OAuth case (test_cloud_oauth_cross_workspace.py) never calls review_rate_limits or get_identity. Add a case there that asks for throttling in the second workspace and asserts the tool is called with that workspace_id (src/prefect_mcp_server/server.py:503).

    prompt coverage: 1 / 1 evals: 6 / 20 eval flaws: 1

    No regressions in the 6 evals that ran. 14 evals couldn't run, so they aren't compared.

    Open the full report for every eval's inputs, outputs and history.

    Some checks were not successful

    1 failing, 2 successful checks

    CI / evals (pull_request) Successful in 4m Details
    Evaluation Results 14 fail, 6 pass in 3m 57s Details
    evalship Check failed: fix 1 eval flaw, or dismiss it in the report. No regressions in the 6 evals that ran. 14 evals couldn't run, so they aren't compared. Details

    Merging is blocked

    The base branch requires all checks to pass.

    Merge pull request
  2. The judge model is swapped

    “fix: replace retired claude-opus-4-1 eval judge model with claude-opus-5”. Every eval passes again. Changing the grader changes what every score means, so evalship's comment says the results may not compare with main's. Still no eval calls the tool with workspace_id, so the flaw stays open and the check stays red.

  3. Every eval passes, the untested fix still blocks

    “style: ruff format after eval model swap”. evalship moves its comment below that push. Merging now takes an eval for the fix, or a member dismissing the flaw. #200 merged on Aug 7.

    github.com/PrefectHQ/prefect-mcp-server/pull/200, after the last push

    zzstoatzz added 1 commit: style: ruff format after eval model swap

    8227588
    evalship-appbot commented 2 minutes ago

    evalship report

    🛑 Check failed: fix 1 eval flaw, or dismiss it in the report.

    Eval main this PR
    🟢 test_agent_correlates_logs_with_rate_limiting ❌ ✅

    ⚖️ Not comparable: core-diagnostics now uses anthropic:claude-opus-5 (evals/conftest.py), so its results won't compare with those on main@2940a3e.

    ⚖️ Scoring changed: this PR edits how core-diagnostics and late-runs decide pass or fail (evals/conftest.py), so their results may not compare with main@2940a3e.

    ⚠️ Eval gap: The new workspace_id parameter on review_rate_limits is only covered by unit tests; the rate-limits-and-cloud OAuth case (test_cloud_oauth_cross_workspace.py) never calls review_rate_limits or get_identity. Add a case there that asks for throttling in the second workspace and asserts the tool is called with that workspace_id (src/prefect_mcp_server/server.py:503).

    prompt coverage: 1 / 1 evals: 20 / 20 eval flaws: 1

    Nothing broke and 1 eval improved.

    Open the full report for every eval's inputs, outputs and history.

    Some checks were not successful

    1 failing, 2 successful checks

    CI / evals (pull_request) Successful in 5m Details
    Evaluation Results All 20 tests pass in 4m 35s Details
    evalship Check failed: fix 1 eval flaw, or dismiss it in the report. Nothing broke and 1 eval improved. Details

    Merging is blocked

    The base branch requires all checks to pass.

    Merge pull request

How we made this

  • Replaying this pull request found four bugs in evalship, fixed before this page went up: a judge's verdict that mentioned "429 warnings" was read as a rate-limited run, a changed tool listed in the review was also reported as untested, the check's title said "No regressions" without saying that most evals couldn't run, and a flaw the reviewer didn't write again on a later push stopped counting, so the check went green with nobody deciding.
  • The eval results are Prefect's own: the JUnit files its CI uploads on every pull request and push to main, read by evalship's JUnit parser. Main's history is the 8 runs from Jul 21 to Aug 4 whose results hadn't expired.
  • The review was written in October 2026, read-only, by evalship's setup prompt (v22) on 2940a3e, the commit #200 branched from, with no access to GitHub or anything after it.
  • Claude Sonnet 5.5 wrote the likely cause and the notes. It answered each push's real PR analysis request in a Claude Code session instead of through the API, and evalship's own validation checked the answer, as it checks every answer. Your pull requests get Claude Haiku 5.5.
  • The comments come from evalship's own code, with the gate setup adds: a new failure, or an eval flaw nobody fixed or dismissed, fails evalship's check.
  • Nothing was posted to PrefectHQ/prefect-mcp-server.

See what evalship finds in your evals

Ask your coding agent: Claude Code Codex Cursor GitHub Copilot

Set up https://evalship.com/setup.md

More case studies: HolmesGPT/holmesgpt · gglucass/nommer-app