Security

Your evals run in your own CI or on your own machine, with your own keys. evalship sees their results, short excerpts of your pull requests, and the diff of a change your coding agent checks before it's committed. Never your secrets or a copy of your repository.

• Your customers' data reaches evalship only if your evals or repository hold it. Owners can stop evalship keeping eval inputs and outputs, and stop it sending anything to a model.
• evalship never copies your repository. It reads files through GitHub while it works and keeps capped excerpts of pull requests: about 16,000 characters, a few hundred lines, per commit it reviews. A change your coding agent checks before it's committed sends its diff, deleted within 7 days.
• A model reads pull requests that touch your prompts or evals, unless you turn it off. It's Claude, through Anthropic's API, which can open the files it needs in that repository; Anthropic doesn't train on them and deletes them within 30 days. Where your evals run on your machines, your coding agent writes the notes instead.
• Your secrets never reach evalship. Your CI or your machine runs the evals with your keys. evalship needs no token or secret from you, and its GitHub App can't read your secrets.
• Only the repositories you pick, read-only. evalship writes nothing but its own pull request comment and check, and each of its GitHub tokens reaches one repository.
• When you leave, your data goes. Remove a repository or uninstall the App, and evalship deletes your eval results, code excerpts and names 30 days later, or when your paid plan ends if that's later. Backups lose them within 3 months.
• A data processing agreement comes with the terms. evalship runs under the GDPR on providers with SOC 2 Type II reports, only its founder can reach your data, and it's working towards SOC 2 itself.

Your customers' data

If your product handles customer data, it can reach evalship four ways:

• Eval results. evalship keeps each eval's input, output and failure message as your eval run writes them, up to 4 KB each, with any metadata the run adds. The model gets the first 600 characters of a newly failing eval's input and output, and 400 of its failure message.
• Eval names and keys, kept as your run writes them.
• Datasets committed to your repository. A pull request's diffs of them go to the model and are kept as excerpts, the model can open them, and a review flaw can quote a line of one.
• Pull request descriptions, kept for the model.

Owners control two switches in the account's settings:

• Eval inputs and outputs. Turned off, evalship keeps no input, output or failure message, and of the metadata only the model under test, and deletes the ones it already had. Pass rates, regressions, flaky evals, the evalship check and untested changes keep working; reports can't show what an eval was asked or answered, or group failures by their message.
• Model notes. Turned off, evalship sends nothing to a model, and keeps no pull request description or eval diff with your runs. PR comments keep results, regressions, flaky evals and untested changes.

Beyond the switches: keep customer data out of eval names, keys and pull request descriptions, and out of the repository, loading it in CI from where it lives. A review flaw that cites a dataset file keeps up to 1,500 characters of a pull request's diff of that file, even with both switches off.

The setup prompt tells your coding agent never to copy customer or personal data into eval cases, the review or a pull request, and never to quote it in a finding.

Your source code

evalship never clones your repository or keeps your files. It reads what it needs through GitHub's API while it processes a run, and keeps only these excerpts, each capped. A change your coding agent checks before it's committed is the one thing it gets from your machine instead: its diff.

In practice it's far below the caps: across the pull requests evalship processed through October 2026, a run kept 16,000 characters at the median and 30,000 at most, mostly the description and the diffs for the model.

What evalship never keeps:

• The files the model opens. They go to the model while it writes its notes, and evalship doesn't store them.
• Artifacts. Each is downloaded to a temporary file, its results read, and the file deleted. Only the results are kept.
• Workflow logs, which evalship never reads.

What goes to a model

Only with Model notes on, and not in a repository whose evals run on its members' machines, nor for a change checked before it's committed: there your coding agent writes the notes, on your own model plan. On a pull request that changes prompts, AI calls or evals, or newly fails evals, a model (Claude Haiku 5.5 today, or Claude Opus when Haiku declines a request, through Anthropic's API) gets the pull request's title and description, the diffs above, excerpts of newly failing evals, and files it opens at the pull request's head commit to check its notes, 300 lines at a time in up to 14 rounds. It can open any file in that repository, and none outside it.

Anthropic doesn't train on these requests and deletes them within 30 days, unless its safety systems flag them. If evalship moves to another model, it goes only through the routes Data and privacy lists, with every field and limit.

Your secrets

• evalship never asks for, receives or stores your API keys or CI secrets. Your workflow runs the evals with its own secrets and uploads a results file; it needs no evalship action, token or secret.
• The GitHub App has no access to secrets, and evalship doesn't read workflow logs.
• Results you send from your own machine carry an upload token evalship makes for you when you approve your coding agent, or in Settings. It only sends results to repositories you can push to, evalship keeps only a hash of it, and you can revoke it in Settings. evalship emails you each time you approve an agent, so one you didn't start shows up.
• The setup prompt tells your coding agent never to read, print, create or change secrets or .env files. When CI needs a key, the agent asks you to add it on GitHub; it checks secret names only.
• The diff of a change checked before it's committed leaves out .env files and keys, and evalship turns away one that holds a file that may hold secrets, keeping none of it.

Access to your repositories

• Only the repositories you choose. Install the App on Only select repositories, and remove one any time in the installation's settings on GitHub. 30 days after you remove a repository, evalship deletes its results, flaws, diffs and name, such as when your work for a client ends.
• Read-only to your code. evalship reads files, pull requests and workflow artifacts, and writes only its own pull request comment and its evalship and evalship notes checks. It can't push, merge or change workflows. GitHub doesn't let an App read only some folders, so read access covers the whole repository. GitHub App permissions lists each permission and why.
• One repository per token. When evalship works on a repository, its GitHub token reaches only that repository. These tokens expire within an hour and live only in the server's memory, never in its database.
• Verified webhooks. evalship acts on a webhook from GitHub only when its signature checks out.
• Forks can't speak for you. Results from a fork's workflow count only for its pull request, never as your default branch's.

Who can see your results

• Private repositories: people GitHub gives access to the repository, and people your account's owners invite, until you uninstall the App. Someone removed on GitHub loses access within 10 minutes. Dismissing flaws and changing settings need push access.
• Invites cover the whole account. Someone an owner invites reads every repository the App is on in that organization, so give each client its own organization. Removing someone from the organization on GitHub ends their invite. Not yet: invites to single repositories.
• Public repositories: anyone, unless you turn off Public dashboard in the repository's settings.
• Coding agents: the setup line on a repository's page lets your agent read its review. The line works for 7 days and until a review is sent, and the agent's setup for 30 days at most.
• evalship staff: today only the founder, through a read-only admin view of stored records that needs a GitHub sign-in, to support you and fix problems.

Where else your data passes

• Webhook queue. GitHub's webhooks carry pull request titles, descriptions and commit messages, never file contents. They wait in a queue in evalship's database until processed and are cleared a day later; one whose processing fails stays until we fix or discard it.
• Logs. Request logs at Railway hold page addresses, which name repositories, pull requests and evals, and IP addresses, for up to 30 days. They leave out the content of webhooks and of the review your agent sends.
• Error reports. Sentry gets stack traces and request metadata naming the account and repository, never request bodies, sign-in links or the contents of database queries.
• Emails. The weekly report and account emails, sent through Resend, name repositories, pull requests and review flaws. They carry no eval inputs, outputs, failure messages or code.
• Backups. Railway backs up the database daily, weekly and monthly, and keeps each backup for up to 89 days. Data evalship deletes is gone from every backup within 3 months.

Sign-in

• Signing in with GitHub gives evalship a token that can do only what both you and the App can, and expires after 8 hours. evalship uses it to check which repositories you can see and whether you own the organization, and keeps it encrypted in the database with keys kept outside it.
• Signing in by email uses a link that works once and expires after 15 minutes. It shows the organizations that invited you; repositories you see through GitHub need a GitHub sign-in, so your organization's two-factor and SSO rules apply.
• A sign-in lasts 30 days at most. Sign out everywhere under Your account ends every session.

Infrastructure

• In transit: HTTPS only, with HSTS, behind Cloudflare. evalship reaches GitHub, Anthropic and its other providers over HTTPS, and its database, which has no public address, over Railway's private network, encrypted with WireGuard.
• Hosting: Railway, in the US. It encrypts the database at rest. Sub-processors lists every provider that handles your data, and Retention how long each kind of data is kept.
• Certified providers: every provider that handles your code or eval results holds a SOC 2 Type II report: GitHub, Cloudflare, Railway, Anthropic, OpenRouter and Microsoft Azure. GitHub, Cloudflare, Anthropic and Azure are ISO 27001 certified too. Sentry and Resend, which get error reports and emails, hold SOC 2 Type II reports as well.
• Our own accounts: only the founder can reach evalship's code, servers and data. GitHub, Cloudflare, Railway, Sentry, Polar, Resend and the founder's email each need a passkey or two-factor to sign in, the evalship GitHub organization requires two-factor, and the founder's laptop disk is encrypted.
• Development: every change to evalship runs Brakeman static analysis and vulnerability audits of its Ruby and JavaScript dependencies in CI. The code had security reviews on October 19 and 29, 2026, and their fixes are deployed.

Compliance

evalship is run by Garm Tech B.V. in the Netherlands, under the GDPR. For personal data in your eval results and repositories you are the controller and evalship the processor, under the data processing agreement, which is part of the terms; nothing to sign.

Have a security questionnaire? Send it to hello@evalship.com and we'll fill it in.

We're working towards a SOC 2 report and a third-party penetration test. If your security review needs either, or ISO 27001 or self-hosting, email hello@evalship.com with what it asks for, and we'll work out with you how to meet it.

Reporting a vulnerability

Email security@evalship.com. If a breach affects your data, we email your account's owners without undue delay, and within 72 hours of learning of it.

Incidents

October 21, 2026: Railway, evalship's host, reported that environment variables of the services it hosts may have been read. We rotated every key and secret evalship uses the next day and found no misuse. No customer accounts existed yet.