Data and privacy
This page lists what evalship stores, what it reads from GitHub, what it sends to a model, and who processes it.
What evalship stores
| Data | Fields |
|---|---|
| Accounts | GitHub account id, login and type, installation id, Polar customer and subscription ids, subscription status, number of paid repositories, the monthly model call count |
| People who sign in | GitHub user id, login, name, avatar URL, email address, GitHub OAuth access and refresh tokens (encrypted at rest), ids of the repositories you can access |
| Repositories | GitHub id, full name, public or private, default branch, badge token, public report setting |
| Pull requests | Number, title, author login, base branch, head commit SHA, state, id of the evalship comment |
| Runs | Commit SHA, branch, baseline, the parsed .evalship.yml, the computed summary, and the changed LLM calls and prompt files: path, line, enclosing function, provider, which cases cover it, and the file's diff, up to 1,500 characters |
| Uploads | Artifact id and name, workflow name, workflow run URL, format, number of cases |
| Case results | Suite, key, name, eval file path, passed, score, threshold, duration, cost, tokens, targets, metadata, and input, output and failure message, each truncated to 4 KB |
| Audits | The audit JSON as your agent sent it, the agent name, the prompt version, a hash of the claim token, who claimed it and when |
| Feedback | Thumbs up and down votes on findings, and thumbs up and down reactions on the evalship PR comment with the GitHub login of the person who reacted |
evalship does not store your source code. It reads files through the GitHub API to find LLM calls and to check audit citations, and keeps only the locations and diff excerpts listed above. Artifacts are downloaded to a temporary file, parsed and deleted.
Case inputs, outputs, failure messages, names and metadata are stored as your eval run writes them. Keep secrets and personal data out of them.
What evalship reads from GitHub
| What | When | Why |
|---|---|---|
| Workflow run artifacts | A workflow run completes | Eval results |
.evalship.yml at the commit |
Each processed run | Configuration |
| Pull request metadata and the merge-base commit | Each pull request run | Baseline and comment |
| The list of changed files and their diffs, and the content of up to 25 changed Python, JavaScript, TypeScript, Ruby, Go and prompt files at the pull request's head | Each pull request run | Finding untested changes |
| Each file cited in an audit's findings, on the default branch | Once, when the audit is claimed | Citation checks |
| Comments and reactions on the pull request | Each comment update, and when the pull request closes | Updating its own comment, and feedback |
What goes to a model
evalship makes one kind of model call: when a run has evals that newly fail, it asks Claude Haiku, through Anthropic's API, to name failure patterns and the likely cause. The request contains:
- up to 20 newly failing cases: their key and suite, the first 600 characters of their input and output, and the first 400 characters of their failure message;
- the diffs of up to 5 changed files with LLM calls or prompts, up to 1,500 characters each.
Nothing is sent for runs without new failures, for private repositories that aren't enabled, or once the account reaches 300 calls in a month. See How PR analysis works.
The audit is written by your own coding agent on your machine, with your model subscription. evalship never runs a model over your repository.
Sub-processors
| Sub-processor | Purpose |
|---|---|
| GitHub | Source of repository data; receives the comments and check runs evalship posts. |
| Railway | Hosting. The application and its database run on Railway. |
| Anthropic | The model call described above. |
| Polar | Merchant of record for billing: payments, invoices and tax. evalship stores only the Polar customer and subscription ids. |
| Sentry | Error tracking. Error reports contain stack traces and request metadata. |
GitHub App permissions
| Permission | Access | Why |
|---|---|---|
| Actions | Read | Download workflow run artifacts. |
| Checks | Read and write | Post the evalship check. |
| Pull requests | Read and write | Read pull requests and their changed files; post and update the comment. |
| Contents | Read | Read .evalship.yml, changed files for untested changes, and the files cited in an audit. |
| Metadata | Read | Required by GitHub for every App: repository names and visibility. |
| Members | Read | Check that the person enabling a private repository or opening billing is an organization owner. |
Retention
| Data | Kept |
|---|---|
| Case results and upload records | One year. Older ones are deleted nightly. |
| Run summaries (pass rates, scores, change counts) | While the App is installed, so trend charts keep working |
| Audits and findings | While the App is installed |
| Unclaimed audits | Stored, but the claim link stops working after 7 days |
When you uninstall the GitHub App, or suspend it, evalship stops processing and posts nothing. All data for that account is deleted 30 days later. If you reinstall within those 30 days, your history is still there.
Deletion requests
To have an account, a repository's data or your user record deleted sooner, email [email protected] with the GitHub account or repository name.