9 prompts without an eval
A change that breaks one of them reaches your users without a failing eval.
Found in 9 places
- src/prefect_mcp_server/server.py:80 get_identity tool description
- src/prefect_mcp_server/server.py:295 get_task_runs tool description
- src/prefect_mcp_server/server.py:387 read_events tool description
- src/prefect_mcp_server/execution_plans.py:187 execution_plans_validate tool description
- src/prefect_mcp_server/execution_plans.py:212 execution_plans_get_schema tool description
- src/prefect_mcp_server/execution_plans.py:262 execution_plans_get tool description
- src/prefect_mcp_server/execution_plans.py:319 execution_plans_publish tool description
- packages/docs_mcp_server/docs_mcp_server/_server.py:86 search_prefect tool description
- plugins/prefect/skills/workflows/SKILL.md:1 workflows skill (hosted plugin instructions)
Fix instructions for your coding agent
Fix this flaw in our LLM evals, found by evalship's review: 9 prompts in our product code have no eval: - src/prefect_mcp_server/server.py:80 (get_identity tool description) - src/prefect_mcp_server/server.py:295 (get_task_runs tool description) - src/prefect_mcp_server/server.py:387 (read_events tool description) - src/prefect_mcp_server/execution_plans.py:187 (execution_plans_validate tool description) - src/prefect_mcp_server/execution_plans.py:212 (execution_plans_get_schema tool description) - src/prefect_mcp_server/execution_plans.py:262 (execution_plans_get tool description) - src/prefect_mcp_server/execution_plans.py:319 (execution_plans_publish tool description) - packages/docs_mcp_server/docs_mcp_server/_server.py:86 (search_prefect tool description) - plugins/prefect/skills/workflows/SKILL.md:1 (workflows skill (hosted plugin instructions)) For each, add cases in a new eval suite that runs on pull requests: inputs that reach the prompt the way production calls it, and checks on what its output must and must not say. Name each eval after the prompt it covers (its function, constant or file), so it's clear which prompt it tests. If one isn't worth an eval, say which and why instead of adding a weak one. Fix it in a pull request; evalship confirms the fix on it.