14 prompts without an eval
A change that breaks one of them reaches your users without a failing eval.
Found in 14 places
- holmes/plugins/prompts/generic_ask_for_issue_conversation.jinja2:1 generic_ask_for_issue_conversation (/api/issue_chat)
- holmes/plugins/prompts/_global_instructions.jinja2:1 _global_instructions
- holmes/plugins/prompts/kubernetes_workload_chat.jinja2:1 kubernetes_workload_chat (/api/workload_health_chat)
- holmes/plugins/prompts/generic_ticket.jinja2:1 generic_ticket (holmes investigate ticket)
- holmes/plugins/toolsets/cilium.yaml:6 cilium/core llm_instructions
- holmes/plugins/toolsets/cilium.yaml:113 hubble/observability llm_instructions
- holmes/plugins/toolsets/openshift.yaml:231 openshift/live-metrics llm_instructions
- holmes/plugins/toolsets/datadog/datadog_general_instructions.jinja2:1 datadog/general instructions
- holmes/plugins/toolsets/datadog/datadog_rds_instructions.jinja2:1 datadog RDS instructions
- holmes/plugins/toolsets/elasticsearch/opensearch_query_assist_instructions.jinja2:1 opensearch query assist instructions
- holmes/plugins/toolsets/elasticsearch/opensearch_ppl_query_docs.jinja2:1 opensearch PPL query docs
- holmes/plugins/toolsets/azure_sql/azure_sql_instructions.jinja2:1 azure/sql instructions
- holmes/plugins/toolsets/atlas_mongodb/instructions.jinja2:1 MongoDB Atlas instructions
- holmes/plugins/toolsets/servicenow_tables/instructions.jinja2:1 servicenow/tables instructions
Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: 14 prompts in our product code have no eval: - holmes/plugins/prompts/generic_ask_for_issue_conversation.jinja2:1 (generic_ask_for_issue_conversation (/api/issue_chat)) - holmes/plugins/prompts/_global_instructions.jinja2:1 (_global_instructions) - holmes/plugins/prompts/kubernetes_workload_chat.jinja2:1 (kubernetes_workload_chat (/api/workload_health_chat)) - holmes/plugins/prompts/generic_ticket.jinja2:1 (generic_ticket (holmes investigate ticket)) - holmes/plugins/toolsets/cilium.yaml:6 (cilium/core llm_instructions) - holmes/plugins/toolsets/cilium.yaml:113 (hubble/observability llm_instructions) - holmes/plugins/toolsets/openshift.yaml:231 (openshift/live-metrics llm_instructions) - holmes/plugins/toolsets/datadog/datadog_general_instructions.jinja2:1 (datadog/general instructions) - holmes/plugins/toolsets/datadog/datadog_rds_instructions.jinja2:1 (datadog RDS instructions) - holmes/plugins/toolsets/elasticsearch/opensearch_query_assist_instructions.jinja2:1 (opensearch query assist instructions) - holmes/plugins/toolsets/elasticsearch/opensearch_ppl_query_docs.jinja2:1 (opensearch PPL query docs) - holmes/plugins/toolsets/azure_sql/azure_sql_instructions.jinja2:1 (azure/sql instructions) - holmes/plugins/toolsets/atlas_mongodb/instructions.jinja2:1 (MongoDB Atlas instructions) - holmes/plugins/toolsets/servicenow_tables/instructions.jinja2:1 (servicenow/tables instructions) For each, add cases in a new eval suite that runs on pull requests: inputs that reach the prompt the way production calls it, and checks on what its output must and must not say. Name each eval after the prompt it covers (its function, constant or file), so it's clear which prompt it tests. If one isn't worth an eval, say which and why instead of adding a weak one. Fix it in a pull request; evalship confirms the fix on it.