Run your Claude evals in CI on your Claude plan

If your product calls Claude and you build it with Claude Code on a Max, Ultra or Team plan, the plan you already pay for can run your evals in CI. It pays in two ways, and each covers different calls: a monthly API credit, and the subscription itself.

What each one pays for

The credit pays for the Claude API and the Agent SDK, and for claude -p run with a key from the organization you link. Runs through the Claude Code GitHub Action count as Claude Code usage, which it doesn't cover (Help Center). A subscription token runs Claude Code in GitHub Actions on your plan's usage instead of API billing (Claude Code docs), but Anthropic's terms send apps that call Claude to API keys (legal and compliance). So your app's calls go through the credit, and the token is for steps that run Claude Code.

The monthly API credit

Free, Pro and Enterprise plans don't include it. The credit arrives each billing cycle, is spent before any credit you've bought, and expires at the end of the cycle (Anthropic).

Claim it

1. From 7 days after you subscribe, open Settings > Billing on claude.ai (on Team, Organization settings > Billing), and in API credits, link a Claude Console organization. A plan links to one organization, and only support can change it, so link the one your app's keys live in.
2. In the Console, make a workspace for CI and give it a spend limit. Every key in the organization draws from the same balance, so the limit keeps evals from spending what your app needs in production.
3. Make a key in that workspace, named after the repository, such as myapp-github-ci, and save it as a secret from your own terminal: gh secret set ANTHROPIC_API_KEY -R you/myapp. It asks for the value, so the key stays out of your shell history.

The workflow

Your evals read the key like any other secret. A paths filter runs them only on pull requests that change what they test, which matters more on a fixed budget:

name: evals
on:
  pull_request:
    paths:
      - "app/llm/**"
      - "prompts/**"
      - "tests/evals/**"
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/evals --junitxml=eval-results.xml -o junit_family=xunit1
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalship-results
          path: eval-results.xml

The push to main gives pull requests results to compare with; leave it unfiltered.

What a run costs

For a suite of 50 test cases, each one call of 3,000 input and 1,000 output tokens (thinking included), at Anthropic's list prices:

A judge that grades each answer is a second call per case: on Haiku 5.5, reading 4,000 tokens and writing 200 per case adds about 3 cents a run. With the paths filter, a project that opens ten pull requests a month that change prompts, with three pushes each, runs the suite about 40 times, merges included: $32 on Sonnet 5.5. Your token counts will differ, and the Console's Cost page shows what your runs spent.

When it runs out

With no other credit in the organization, calls stop until next month's credit arrives, and nothing is charged to your plan. Each one fails with "Your credit balance is too low to access the Anthropic API". To keep going, buy credits or turn on auto-reload on the Console's Billing page. Unused credit doesn't roll over.

Your subscription, for Claude Code steps

Some eval steps run Claude Code itself: an LLM judge called through claude -p, or evals of a skill, plugin or subagent, where Claude Code is what you ship. Those can run on your subscription, Pro included:

1. In your terminal, run claude setup-token. It prints a long-lived token for your subscription.
2. Save it: gh secret set CLAUDE_CODE_OAUTH_TOKEN -R you/myapp.
3. Pass it to the step:

      - run: npm install -g @anthropic-ai/claude-code
      - run: python tests/evals/judge.py # runs claude -p --model claude-opus-5-5 on each answer
        env:
          CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}

With the Claude Code GitHub Action, the same secret goes in its claude_code_oauth_token input.

Before you do:

• It shares your limits. CI runs draw from the same usage limits as your own Claude Code sessions, so a long eval run leaves less for your coding.
• It's yours alone. The token is tied to the subscription of whoever made it. Keep it a repository secret: for a secret that several repositories or people share, Anthropic recommends an API key (GitHub Actions docs).
• It's for Claude Code, not your app. Anthropic's terms keep subscription sign-in for ordinary use of Claude Code and Anthropic's own apps, and ask developers whose products call Claude to use API keys. Your app's calls take the credit's key, in CI too.

evalship's own evals work this way: the code under test calls the API with a key, and the judge runs Claude Opus 5.5 through claude -p on a subscription token.

Which one to use

• Your evals call your app's prompts: the API credit. On Max, it pays for about 125 runs of the Sonnet 5.5 suite above.
• You're on Pro: there's no credit. The subscription covers Claude Code steps, and your app's calls need API credit you buy.
• You need more than the credit: run evals only when prompts change, grade with Haiku 5.5, or buy credits on top. The included credit is still spent first.

How evalship does this

evalship reads the results file your workflow uploads and compares every pull request with your default branch, eval by eval. Evals that the API turned away for a low credit balance show as couldn't run, not as failures, and the pull request comment says the credit ran out. API keys has the details.

evalship is free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart). When your evals call Claude, the agent points you to the credit as it asks for the key.

Further reading

• Anthropic, API credits for Max, Ultra, and Team plans: amounts, claiming, and what the credit covers.
• Claude Code, GitHub Actions: the action, claude setup-token and subscription tokens.
• How noisy are LLM evals in CI?: how often passing evals fail by chance, measured on public repositories.