Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

Case study · HolmesGPT/holmesgpt

A prompt cut that HolmesGPT's own eval check let through

HolmesGPT is an open-source SRE agent, a CNCF Sandbox project. In January 2026, pull request #1452 commented out parts of its prompt to see which ones mattered. One eval broke, and HolmesGPT's eval check stayed green.

230 / 230
runs of 162_get_runbooks passed in January before #1452
3 of 3
pushes evalship's check would have blocked
0 of 3
pushes HolmesGPT's own eval check failed
4
flaws evalship's review found in its evals

What evalship's review found

At setup, your coding agent reads every scorer, judge and prompt, and writes down where the evals fall short. On HolmesGPT/holmesgpt at f940832, the commit #1452 branched from, it found:

1 Eval gapUntested prompt

14 prompts without an eval

A change that breaks one of them reaches your users without a failing eval.

Found in 14 places
  • holmes/plugins/prompts/generic_ask_for_issue_conversation.jinja2:1 generic_ask_for_issue_conversation (/api/issue_chat)
  • holmes/plugins/prompts/_global_instructions.jinja2:1 _global_instructions
  • holmes/plugins/prompts/kubernetes_workload_chat.jinja2:1 kubernetes_workload_chat (/api/workload_health_chat)
  • holmes/plugins/prompts/generic_ticket.jinja2:1 generic_ticket (holmes investigate ticket)
  • holmes/plugins/toolsets/cilium.yaml:6 cilium/core llm_instructions
  • holmes/plugins/toolsets/cilium.yaml:113 hubble/observability llm_instructions
  • holmes/plugins/toolsets/openshift.yaml:231 openshift/live-metrics llm_instructions
  • holmes/plugins/toolsets/datadog/datadog_general_instructions.jinja2:1 datadog/general instructions
  • holmes/plugins/toolsets/datadog/datadog_rds_instructions.jinja2:1 datadog RDS instructions
  • holmes/plugins/toolsets/elasticsearch/opensearch_query_assist_instructions.jinja2:1 opensearch query assist instructions
  • holmes/plugins/toolsets/elasticsearch/opensearch_ppl_query_docs.jinja2:1 opensearch PPL query docs
  • holmes/plugins/toolsets/azure_sql/azure_sql_instructions.jinja2:1 azure/sql instructions
  • holmes/plugins/toolsets/atlas_mongodb/instructions.jinja2:1 MongoDB Atlas instructions
  • holmes/plugins/toolsets/servicenow_tables/instructions.jinja2:1 servicenow/tables instructions
Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: 14 prompts in our product code have no eval:
- holmes/plugins/prompts/generic_ask_for_issue_conversation.jinja2:1 (generic_ask_for_issue_conversation (/api/issue_chat))
- holmes/plugins/prompts/_global_instructions.jinja2:1 (_global_instructions)
- holmes/plugins/prompts/kubernetes_workload_chat.jinja2:1 (kubernetes_workload_chat (/api/workload_health_chat))
- holmes/plugins/prompts/generic_ticket.jinja2:1 (generic_ticket (holmes investigate ticket))
- holmes/plugins/toolsets/cilium.yaml:6 (cilium/core llm_instructions)
- holmes/plugins/toolsets/cilium.yaml:113 (hubble/observability llm_instructions)
- holmes/plugins/toolsets/openshift.yaml:231 (openshift/live-metrics llm_instructions)
- holmes/plugins/toolsets/datadog/datadog_general_instructions.jinja2:1 (datadog/general instructions)
- holmes/plugins/toolsets/datadog/datadog_rds_instructions.jinja2:1 (datadog RDS instructions)
- holmes/plugins/toolsets/elasticsearch/opensearch_query_assist_instructions.jinja2:1 (opensearch query assist instructions)
- holmes/plugins/toolsets/elasticsearch/opensearch_ppl_query_docs.jinja2:1 (opensearch PPL query docs)
- holmes/plugins/toolsets/azure_sql/azure_sql_instructions.jinja2:1 (azure/sql instructions)
- holmes/plugins/toolsets/atlas_mongodb/instructions.jinja2:1 (MongoDB Atlas instructions)
- holmes/plugins/toolsets/servicenow_tables/instructions.jinja2:1 (servicenow/tables instructions)
For each, add cases in a new eval suite that runs on pull requests: inputs that reach the prompt the way production calls it, and checks on what its output must and must not say. Name each eval after the prompt it covers (its function, constant or file), so it's clear which prompt it tests. If one isn't worth an eval, say which and why instead of adding a weak one.
Fix it in a pull request; evalship confirms the fix on it.
2 ImportantEval bug

A pull request that breaks a toolset is reported as a setup failure, not a failed eval

Example: a PR that renames a BashExecutorConfig field, breaks bash_instructions.jinja2, or breaks the Loki health check (toolset_grafana_loki.py:71-93) silently disables that toolset for users. In the eval, all 9 PR regression cases become '🚧 setup failure' under a '✅ Results of HolmesGPT evals' header with no failure warning, and the benchmark score is unchanged. Toolset config changes like the recent claude/standardize-toolset-config-fields branch are exactly this kind of PR.

tests/llm/utils/default_toolsets.yaml:23
bash:

⚠ bash, kubernetes/core, kubernetes/logs, helm/core, internet, robusta and others are enabled for every case, so their product prerequisite checks run in every eval

holmes/plugins/toolsets/bash/bash_toolset.py:345
self.config = BashExecutorConfig(**config)

⚠ product code in bash's prerequisite; if it raises, holmes/core/tools.py:835-837 marks the toolset FAILED

tests/llm/utils/mock_toolset.py:810
raise ToolsetPrerequisiteError(

⚠ the harness turns any FAILED toolset that the config enables into this error

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: A pull request that breaks a toolset is reported as a setup failure, not a failed eval. Example: a PR that renames a BashExecutorConfig field, breaks bash_instructions.jinja2, or breaks the Loki health check (toolset_grafana_loki.py:71-93) silently disables that toolset for users. In the eval, all 9 PR regression cases become '🚧 setup failure' under a '✅ Results of HolmesGPT evals' header with no failure warning, and the benchmark score is unchanged. Toolset config changes like the recent claude/standardize-toolset-config-fields branch are exactly this kind of PR. See tests/llm/utils/default_toolsets.yaml:23, holmes/plugins/toolsets/bash/bash_toolset.py:345, and tests/llm/utils/mock_toolset.py:810. In property_manager.handle_test_error, count ToolsetPrerequisiteError as a regression instead of a setup failure, and have the PR comment show a failure warning whenever setup failures are above zero.
Fix it in a pull request; evalship confirms the fix on it.
3 ImportantEval bug

The judge grades every earlier assistant message, including fixture history, as part of the answer

An expected element stated in any mid-investigation message counts even when the final answer drops it, contradicts it, or is empty. For example, in 61_exact_match_counting (a PR regression case), a step saying 'test-61 has 6 running pods' followed by a final answer of '15 pods' still matches 'either 6, 6 pods'. In the 5 runnable multi-turn cases with assistant history (44, 92, 93, 161, 163), the fixture's prior turns (or a summary of them) are graded as part of the answer.

tests/llm/utils/property_manager.py:137
include_intermediate = request.config.getoption("include_intermediate", True)

⚠ on by default; neither eval-regression.yaml:811 nor run_benchmarks_local.py passes --no-include-intermediate

holmes/core/tool_calling_llm.py:534
messages=messages,

⚠ result.messages is the full list, including the conversation_history the fixture supplied

tests/llm/utils/property_manager.py:156
evaluation_output = "## Intermediate LLM Outputs:\n\n"

⚠ every assistant message except the final one goes to the judge

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: The judge grades every earlier assistant message, including fixture history, as part of the answer. An expected element stated in any mid-investigation message counts even when the final answer drops it, contradicts it, or is empty. For example, in 61_exact_match_counting (a PR regression case), a step saying 'test-61 has 6 running pods' followed by a final answer of '15 pods' still matches 'either 6, 6 pods'. In the 5 runnable multi-turn cases with assistant history (44, 92, 93, 161, 163), the fixture's prior turns (or a summary of them) are graded as part of the answer. See tests/llm/utils/property_manager.py:137, holmes/core/tool_calling_llm.py:534, and tests/llm/utils/property_manager.py:156. Grade only result.result: pass --no-include-intermediate in eval-regression.yaml and run_benchmarks_local.py (or flip the default). If you keep intermediate steps, exclude messages that came from test_case.conversation_history and tell the judge to score only the '## Final Output' section.
Fix it in a pull request; evalship confirms the fix on it.
4 ImportantEval bug

The investigate suite never fails when a section that should be missing is present

Suppose investigate_issues stopped honouring the client's requested sections. Structured output is already off for bedrock and gemini models (investigation.py:116-118), and the model could return all 7 DEFAULT_SECTIONS. 01_oom_kill would still pass. Even when a section assert does fail, the /eval comment and benchmark tables show the case as passed.

tests/llm/fixtures/test_investigate/01_oom_kill/test_case.yaml:10
Key Findings: False

⚠ the only case that requests custom sections expects 4 default sections to be absent

tests/llm/test_investigate.py:240
if expected_section_array_content:

⚠ False entries are skipped, so absence is never checked deterministically

tests/llm/test_investigate.py:196
scores["sections"] = sections_eval.score

⚠ the sections judge's score is computed but never asserted; only correctness is (line 232)

Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: The investigate suite never fails when a section that should be missing is present. Suppose investigate_issues stopped honouring the client's requested sections. Structured output is already off for bedrock and gemini models (investigation.py:116-118), and the model could return all 7 DEFAULT_SECTIONS. 01_oom_kill would still pass. Even when a section assert does fail, the /eval comment and benchmark tables show the case as passed. See tests/llm/fixtures/test_investigate/01_oom_kill/test_case.yaml:10, tests/llm/test_investigate.py:240, and tests/llm/test_investigate.py:196. In test_investigate.py, assert that each False entry in expected_sections is absent from result.sections (or empty), and report the pytest outcome rather than the judge score as the case's pass status.
Fix it in a pull request; evalship confirms the fix on it.

What evalship would have said on #1452

  1. The first push breaks an eval

    “WIP minimize system prompt” comments out the system prompt and the runbook context. Without its runbook catalog, the agent's answer misses what the runbook says, and 162_get_runbooks fails. evalship compares it with the last 30 runs on master, where it always passed, calls it a regression and names the likely cause. Its reviewer also reads what the pull request does to the evals themselves: the kubectl toolsets are commented out of every case, so the results won't compare with master. With less prompt to send, the eval run cost less than half as much.

    HolmesGPT's eval workflow reports but never fails the build, so its check stays green. evalship's fails and blocks the merge.

    github.com/HolmesGPT/holmesgpt/pull/1452

    evalship-appbot commented 2 minutes ago

    evalship report

    🛑 Check failed: 1 eval went from passing to failing; fix 1 eval flaw, or dismiss it in the report.

    Eval master this PR
    🔴 162_get_runbooks ✅ ❌

    🔍 Likely cause: the system prompt and runbook context commented out in build_initial_ask_messages in holmes/core/prompt.py.

    ⚖️ Scoring changed: this PR changes the test cases or expected answers of ask-holmes in the same PR as the prompt it grades (tests/llm/fixtures/test_ask_holmes/101_loki_historical_logs_pod_deleted/toolsets.yaml), so its results may not compare with master@f940832.

    🔍 Before you merge: build_initial_ask_messages no longer sends the system prompt, the TodoWrite reminder or the runbook context, and build_chat_messages drops its system prompt and runbook context too, so ask-holmes and fast-benchmark grade a model with no Holmes instructions. This looks like leftover debugging; restore these lines before reading any result (holmes/core/prompt.py:122).

    ⚖️ Not comparable: Commenting out kubernetes/core and kubernetes/logs in the default toolsets, plus the same in cases 101 and 179, changes the tools every ask-holmes case runs with, so results won't compare with master. Revert, or land the toolset change separately (tests/llm/utils/default_toolsets.yaml:3).

    ⚠️ Eval note: get_historical_metrics now always returns early with no metrics because the build_historical_metrics call is commented out, and case 43 loses its regression tag. Restore both or the regression run and its timing history silently change (tests/llm/utils/braintrust_history.py:355).

    💰 Eval run cost -56% ($1.99 to $0.88).

    prompt coverage: 1 / 1 evals: 7 / 8 eval flaws: 1

    1 regression in ask_holmes, likely from the system prompt and runbook context commented out in build_initial_ask_messages in holmes/core/prompt.py.

    Open the full report for every eval's inputs, outputs and history.

    Some checks were not successful

    1 failing, 1 successful checks

    Run eval regression tests / llm_evals (pull_request) Successful in 6m Details
    evalship Check failed: fix 1 eval flaw, or dismiss it in the report. 1 regression in ask_holmes, likely from the system prompt and runbook context commented out in build_initial_ask_messages in holmes/core/prompt.py. Details

    Merging is blocked

    The base branch requires all checks to pass.

    Merge pull request
  2. A merge from master changes nothing

    162_get_runbooks still fails. evalship moves its comment below the new commit, and the merge stays blocked.

  3. The eval passes, two leftovers still block

    “wip conrtol prompting parts” puts the system prompt and runbook context back, behind a switch that's on by default. All 8 evals pass. Two test setup edits are still in the pull request: a stub that switches off HolmesGPT's own comparison with history, and a regression tag commented out. Each is an eval flaw, so evalship's check stays red until someone fixes it or dismisses it in evalship's report. Merging with them is then a choice someone made. #1452 merged on Feb 3.

    github.com/HolmesGPT/holmesgpt/pull/1452, after the fix

    Sheeproid added 1 commit: wip conrtol prompting parts

    5110d59
    evalship-appbot commented 2 minutes ago

    evalship report

    🛑 Check failed: fix 2 eval flaws, or dismiss them in the report.

    ⚠️ Eval note: if True: now returns before the historical metrics are computed, so the code below it is dead and runs always report no historical metrics. This looks like a debugging leftover; restore the original if not metrics: logic (tests/llm/utils/braintrust_history.py:361).

    ⚠️ Test cases changed: The regression tag on 43_current_datetime_from_prompt is commented out in the same PR that changes how the prompt is built, so the CI -m regression run drops the datetime case. Keep the tag so this change is still graded on it (tests/llm/fixtures/test_ask_holmes/43_current_datetime_from_prompt/test_case.yaml:10).

    prompt coverage: 4 / 4 evals: 8 / 8 eval flaws: 2

    Open the full report for every eval's inputs, outputs and history.

    Some checks were not successful

    1 failing, 1 successful checks

    Run eval regression tests / llm_evals (pull_request) Successful in 8m Details
    evalship Check failed: fix 2 eval flaws, or dismiss them in the report. All evals pass. Details

    Merging is blocked

    The base branch requires all checks to pass.

    Merge pull request

How we made this

  • The eval results are HolmesGPT's own. Its eval bot posts each push's results in one pull request comment, and GitHub keeps every earlier version of that comment. The Actions logs and artifacts have expired.
  • Master's history is a stand-in. Master's own per-eval results expired too. In their place are the 30 runs, from 16 pull requests between Jan 25 and Jan 29, that changed nothing in the agent's code (holmes/), so they ran the same agent as master.
  • Replaying this pull request found three bugs in evalship, fixed before this page went up: commented-out code didn't count as a change, a changed prompt file wasn't matched to the suite that runs a function in it, and eval cases kept in a folder named after their test file weren't treated as that suite's own.
  • The review was written in October 2026, read-only, by evalship's setup prompt (v22) on f940832, the commit #1452 branched from, with no access to GitHub or anything after it.
  • Claude Sonnet 5.5 wrote the likely cause and the notes. It answered each push's real PR analysis request in a Claude Code session instead of through the API, and evalship's own validation checked the answer, as it checks every answer. Your pull requests get Claude Haiku 5.5.
  • The comments come from evalship's own code, with the gate setup adds: a new failure, or an eval flaw nobody fixed or dismissed, fails evalship's check.
  • Nothing was posted to HolmesGPT/holmesgpt.

See what evalship finds in your evals

Ask your coding agent: Claude Code Codex Cursor GitHub Copilot

Set up https://evalship.com/setup.md

More case studies: PrefectHQ/prefect-mcp-server · gglucass/nommer-app