Monitor

evalship checks every AI call your app makes in production and emails you the moment one fails, promises something, shares private data or frustrates a user.

Connect production

1. Open your repository's Monitor page on evalship and click Make a production key. evalship shows the key once.
2. Send your AI calls one of three ways:
   • Your coding agent. Paste the line the page gives you, Send this app's AI calls to evalship: https://evalship.com/production.md?repo=OWNER/REPO. The agent finds your LLM calls, names each one, and adds a call after each with the HTTP client your app already has.
   • OpenRouter. In OpenRouter's Settings, under Broadcast, add a Webhook destination with the URL https://evalship.com/api/v1/otlp/v1/traces and the header Authorization: Bearer <your key>. Name each call with trace: { call: "support-chat" } in its request.
   • OpenTelemetry. Point an OTLP/HTTP exporter (http/protobuf or http/json) at the same URL, with the same header. If the app already exports elsewhere, add a second exporter in code: OTEL_EXPORTER_OTLP_* variables would change both. Set evalship.call to the call's name on each LLM span. evalship keeps only the spans of AI calls, tools and agents, those with gen_ai.*, llm.* or evalship.* attributes, so the app's other spans cost nothing.
3. Add the key to production's environment as EVALSHIP_PRODUCTION_KEY and deploy. Calls show on Monitor within a minute of the first one.

Making a new key replaces the old one: calls with the old key stop arriving.

What it flags

Rules check every call. A model (Claude Haiku) checks what only reading can tell, and its flag stands only when it quotes the call word for word. When unsure, it flags nothing: a wrong alert costs more than a missed one.

Calls with the same flag in the same AI call make one issue. evalship emails your organization's owners when an issue opens, at most once every 10 minutes per repository. An issue you resolve opens again when a new call has it. An issue you ignore keeps counting, without emails.

Checks

An AI call whose answer reaches a user gets the reply checks above without any setup. For a call whose output feeds code, such as classifying or extracting, list what a good answer must be in .evalship.yml under monitor, by the call's name:

monitor:
  classify-ticket:
    - json
    - matches: '"category": "(billing|bug|other)"'
  support-chat:
    - replies
    - judge: "Does the answer stay on the user's question?"

Naming a call's checks replaces the reply checks for that call, so list replies too to keep them. Each call takes up to 20 checks. evalship reads .evalship.yml from the default branch, within 10 minutes of a change.

Agents

An agent's run is a trace: each step a span under its parent, AI calls, tool calls and anything else. OpenTelemetry sends them as they are. From your own code, give every span of one run the same trace_id, its own span_id, and its parent's span_id as parent_span_id, and send tool calls as "kind": "tool". Monitor shows each trace as a tree, with the steps that failed or were flagged marked.

The API

POST https://evalship.com/api/v1/calls with Authorization: Bearer <your key> takes one call, or up to 100 as {"calls": [...]}, and answers 202 with {"received": n}.

A call evalship can't keep answers 422 with what's wrong, and nothing of its list is kept. Sending the same span_id twice keeps it once. OpenTelemetry takes up to 1,000 spans a request: past that, the answer's partialSuccess says how many it left out. A repository locked by its plan answers 403.

Limits

Calls count as they arrive. Past the first, the API answers 429 until the next month. Past the second, the rules still check every call. Input and output are kept up to 20,000 characters each, for as long as your plan keeps eval history.