# Set up evalship (setup prompt v3)

You are setting up evalship (https://evalship.com) in this repository. evalship reads LLM eval results from GitHub Actions and comments on every pull request with what changed and why.

Start by telling the user, in four short lines, what you are about to do:
1. Find the evals and how CI runs them (read only).
2. Make CI save the eval results for evalship (a small workflow change, in a pull request).
3. Review the evals and note what could be better (read only).
4. Send the review to evalship and hand over a link.

Then work through the steps. Ask the user only where a step says ASK. Keep messages short.

Ground rules:
- Never read, print, create or change secrets (.env files, keys, credentials). If CI needs one, name it and give the user the `gh secret set NAME` command.
- Never weaken, skip or delete existing evals or assertions, and never change product code.
- Every claim about the repo must cite a file and line you read.
- evalship supports GitHub Actions only. Find OWNER/REPO with `git remote get-url origin`.

## Step 1: Find the evals (read only)
Look for: promptfoo (promptfooconfig.*), Evalite (*.eval.ts), vitest-evals (describeEval), DeepEval (deepeval, assert_test), Braintrust (Eval( from braintrust), LangSmith (evaluate( with langsmith), Inspect AI (@task), OpenAI Evals (evals/registry), Ragas, Opik, Langfuse experiments, and hand-rolled evals (evals/, test/evals/, spec/evals/, tests that call an LLM SDK and assert on its output). Also find the prompts and LLM calls they cover (prompt files and inline prompts next to OpenAI, Anthropic, Google, LiteLLM, LangChain, Vercel AI SDK or similar calls), and which GitHub Actions workflows run the evals.
Tell the user what you found in at most three lines. If there are no evals, say so, skip to step 3 and write findings about what should be tested first.

## Step 2: Connect CI
Goal: the workflow that runs the evals triggers on `pull_request` and on `push` to the default branch, writes a results file, and uploads it as an artifact named `evalship-results`. No evalship action, token or secret is needed.
a. Results file: JUnit XML (pytest `--junitxml=eval-results.xml -o junit_family=xunit1`, vitest `--reporter=junit --outputFile=eval-results.xml`, jest-junit, rspec_junit_formatter, go-junit-report), promptfoo JSON (`promptfoo eval -o eval-results.json`) or evalship JSON (https://evalship.com/schema/v1.json). In pytest, add `record_property("score", s)` and `record_property("threshold", t)` where an eval computes them.
b. Right after the eval step:
```yaml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml
```
c. If no workflow runs the evals, ASK before creating `.github/workflows/evals.yml`.
d. Show the diff and ASK before committing. Commit on a branch named `evalship/setup` and open a pull request with `gh pr create` (or show the commands). Change nothing else.
e. If the evals call a paid API in CI, list the secrets they need with their `gh secret set` commands.

## Step 3: Review the evals (read only)
Write at most 8 findings, most valuable first. Each is specific to this repo, cites files and lines, and ends in one concrete action. Generic advice is not a finding; if you see two real issues, report two. Kinds:
- not_in_ci: evals exist but CI never runs them, or not on pull requests
- untested_prompt: a prompt or LLM call no eval exercises
- untested_instruction: a requirement in a prompt ("never promise refunds above 100 euros") that no case checks
- judge_only: every assertion is LLM-as-judge; name deterministic checks that would catch the same failures
- too_easy: cases that can't realistically fail
- thin_dataset: few cases, near-duplicates, no edge or adversarial cases, one language while the prompt handles many
- cost: expensive suites, repeated model calls, no caching, a big judge model where a small one would do
- duplicated_setup: two frameworks testing the same thing, dead eval files
- nondeterminism: temperature above 0 without repeats or tolerances, live network calls, no seeds
Also write 3 to 6 upgrade steps (plan) toward: evals on every PR, fast and cheap, deterministic checks before judges, stable case names. Mark what step 2 already did as "applied", the rest as "proposed". Change nothing in this step.

## Step 4: Send the review
Write it as JSON to a temporary file outside the repo, following https://evalship.com/schema/audit-v1.json:
```json
{
  "schema": "evalship/audit-v1", "repository": "OWNER/REPO", "agent": "claude-code", "prompt_version": "3",
  "headline": "One or two sentences on the state of this repo's evals.",
  "suites": [{ "name": "support-bot", "framework": "pytest", "path": "tests/evals/test_support_bot.py", "cases": 12,
    "deterministic_assertions": 12, "judge_assertions": 0, "exercises": ["app/support_bot.py#answer"], "models": ["gpt-4o-mini"],
    "command": "pytest tests/evals", "ci_triggers": ["pull_request", "push"], "results_format": "junit", "est_cost_per_run_usd": 0.02 }],
  "prompts": [{ "path": "app/support_bot.py", "line": 3, "symbol": "SYSTEM_PROMPT", "tested_by": ["support-bot"] }],
  "findings": [{ "kind": "untested_instruction", "severity": "important", "title": "Nothing checks the refund limit",
    "evidence": [{ "file": "app/support_bot.py", "line": 4, "quote": "Never promise refunds above 100 euros", "note": "no case asks for a large refund" }],
    "why_it_matters": "One sentence.", "action": "One concrete step." }],
  "plan": [{ "title": "Run evals on pull requests", "detail": "One sentence.", "status": "applied" }],
  "ci": { "workflow": ".github/workflows/evals.yml", "uploads_results": true, "artifact_name": "evalship-results", "setup_pr_url": "https://github.com/OWNER/REPO/pull/1", "secrets_needed": [] },
  "badge_added": false
}
```
Severity is info, notice or important; plan status is applied, proposed or skipped; quotes are copied exactly from the cited line. Show the findings to the user and ASK before sending. Then run:
`curl -sS -X POST 'https://evalship.com/api/v1/audits' -H 'Content-Type: application/json' --data @/tmp/evalship-audit.json`
If it returns an error, fix the JSON and retry. The response has a `url`. Tell the user: "Open <url> to connect this repo to evalship and see the review. Merge the setup PR and evalship will comment on your next pull request."

## After that (optional)
Offer these in one line each and do only what the user picks:
- Apply any proposed upgrade steps now.
- Add the evalship badge to the README, next to the existing badges (or under the title if there are none): `[![evalship](https://evalship.com/gh/OWNER/REPO/badge.svg)](https://evalship.com/gh/OWNER/REPO)` for a public repo. For a private repo the badge needs a token: say the snippet is on the repo's Setup tab in evalship.