# evalship

> evalship tests, monitors and optimizes every AI call your product makes. Get notified whenever your AI fails, and see what's wrong and how to fix it.

In production, your app sends evalship its AI calls, from its own code, through OpenRouter's Broadcast or as OpenTelemetry traces: evalship checks each one, groups what's wrong into issues and emails the moment one opens, with the calls behind it and a line for your coding agent to fix it. Your eval workflow keeps running in your own GitHub Actions with your own keys, and uploads a results file from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust or LangSmith (JUnit XML, promptfoo JSON, pytest-json-report JSON, Vitest or Jest JSON, an Inspect AI log or evalship JSON) that evalship reads when the run finishes. Teams that keep model keys off GitHub run the evals on their own machines instead, and their coding agent sends the results. Setup is a prompt for your coding agent: it reviews your evals, wires them into CI or your machine and opens a setup pull request. Before the agent finishes a change to a prompt, model or AI call, it runs the evals the change affects and sends evalship the diff with the results. Optimize has the agent try prompt and model changes on your machine, and evalship says which win beyond chance. evalship works next to eval platforms such as Langfuse, Braintrust and LangSmith rather than replacing them. Public repositories are free.

## Product

- [Detect errors in your AI calls](https://evalship.com/product/monitor): Every AI call your app makes in production, checked the moment it comes in, with an email when one fails, promises something or shares private data.
- [Review your evals](https://evalship.com/product/review): At setup, your coding agent reviews every scorer, judge and prompt in your LLM evals and finds their flaws, each with a fix.
- [Pull request reports](https://evalship.com/product/pull-requests): On every pull request, the LLM evals the change really broke and the likely cause, with flaky evals set aside and untested AI calls flagged.
- [Optimize prompts and models](https://evalship.com/product/optimize): Your coding agent tries prompt and model changes on your machine, and evalship says which win beyond chance, and by how much.
- [Eval runs and weekly report](https://evalship.com/product/test): Every eval run from GitHub and your engineers' machines with what to do next, the week's insights, and a Monday email listing what needs your team.
- [How it works](https://evalship.com/product/how-it-works): Support bot ELIZA from setup on: evalship finds the flaws in its evals, blocks a pull request that breaks a rule, then finds a better prompt.
- [Integrations](https://evalship.com/product/integrations): evalship reads results from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust and LangSmith, next to the eval platform you keep.
- [evalship for coding agents](https://evalship.com/product/agents): Before your coding agent finishes a change to a prompt or model, it runs the evals before and after, and evalship says if it got worse. No account.

## Docs

- [Quickstart](https://evalship.com/docs/quickstart.md): evalship works in two parts. Setup reviews your evals and shows what they miss, with a fix for each flaw.
- [Monitor](https://evalship.com/docs/monitor.md): evalship checks every AI call your app makes in production and emails you the moment one fails, promises something, shares private data or frustrates a user.
- [What your agent does](https://evalship.com/docs/agent.md): Setup is a prompt your coding agent follows in your repository.
- [The review](https://evalship.com/docs/audit.md): The review is your coding agent's look at your evals, written while it follows the setup prompt.
- [What evalship checks](https://evalship.com/docs/checks.md): evalship checks your evals twice. Once at setup, the review asks whether your evals can catch a bad change at all.
- [How the PR comment works](https://evalship.com/docs/comparison.md): When an eval workflow finishes, evalship stores its results as a run for that commit, compares the run with a baseline, and reports the difference in a comment…
- [Untested changes](https://evalship.com/docs/untested-changes.md): On every pull request, evalship finds the AI calls and prompt files the pull request adds or changes, and checks whether any eval exercises them.
- [Weekly report](https://evalship.com/docs/weekly-report.md): Every Monday at 8:00 UTC, evalship emails each account's weekly report to everyone who reads its reports: the people the account invited, and GitHub users with…
- [Optimize](https://evalship.com/docs/optimize.md): Your coding agent finds changes your evals can prove better, runs them on your machine, and evalship says which win beyond chance.
- [Badges](https://evalship.com/docs/badges.md): evalship serves README badges with live eval results from your default branch.
- [Integrations](https://evalship.com/docs/integrations.md): evalship reads the results your evals already write, from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust or LangSmith.
- [Connecting CI](https://evalship.com/docs/ci.md): evalship reads eval results from GitHub Actions workflow artifacts.
- [Result formats](https://evalship.com/docs/formats.md): evalship reads six result formats. Each result becomes the same eval record: a suite, a key, pass or fail, and optionally a score, threshold, duration, cost,…
- [Configuration](https://evalship.com/docs/configuration.md): evalship works without configuration. To change its behavior, add .evalship.yml to the root of your repository.
- [Pricing and billing](https://evalship.com/docs/pricing.md): evalship's plans and prices, the launch offer, how users are counted, additional usage, trials and billing.

## About evalship

- [FAQ](https://evalship.com/faq.md): Common questions about evalship: running evals, frameworks, CI, coding agents, privacy, gating, and using it next to Langfuse, Braintrust or LangSmith.
- [Security](https://evalship.com/security.md): What evalship keeps of your customers' data and source code, what goes to a model, how it handles your secrets, and how it reaches your repositories.
- [Data and privacy](https://evalship.com/privacy.md): This page lists what evalship stores, what it reads from GitHub, what it sends to a model, and who processes it.
- [Data processing agreement](https://evalship.com/dpa.md): This agreement is part of the terms between you and Garm Tech B.V. ("we", "us"), and covers the personal data evalship processes on your behalf.
- [Terms of service](https://evalship.com/terms.md): The terms for using evalship.com and the evalship GitHub App.

## Articles

- [Recorded LLM replies in Rails tests pass any prompt change](https://evalship.com/articles/recorded-llm-replies-in-rails-tests): In 16 of 19 Ruby apps that replay their LLM calls in tests, the replay ignores the request, so any prompt or model change passes.
- [LLM evals in Ruby, Go and Rust, without Python or Node](https://evalship.com/articles/llm-evals-in-ruby-go-rust): Run LLM evals in a Ruby, Go or Rust app without adding Node or Python to CI: call your real code, check in code first, run each test case 3 times.
- [LLM evals that can't fail: flaws in 10 open-source eval suites](https://evalship.com/articles/llm-eval-flaws): Every one of 10 open-source LLM eval suites had a test case that couldn't fail. The flaws we found most, with examples and how to check yours.
- [LLM evals on pull requests: how the tools compare (2026)](https://evalship.com/articles/llm-evals-on-pull-requests): promptfoo, Braintrust, Langfuse, LangSmith, DeepEval and Raindrop on pull requests: what each compares with main, and what fails the check.
- [LLM as a judge: five mistakes in real eval suites](https://evalship.com/articles/llm-as-a-judge-mistakes): In 6 of 9 open-source eval suites, the LLM judge had a flaw, most often grading its own model's answers. Five mistakes, and how to fix each.
- [Run your Claude evals in CI on your Claude plan](https://evalship.com/articles/claude-evals-on-your-plan): Claude Max, Ultra and Team plans include $100 to $500 a month of API credit. Pay for your evals in CI with it, or with your subscription.
- [Langfuse experiments on every pull request](https://evalship.com/articles/langfuse-experiments-in-ci): Run Langfuse experiments on every pull request and see which test cases the change broke, compared with your default branch.
- [Braintrust evals on every pull request](https://evalship.com/articles/braintrust-evals-in-ci): Run Braintrust evals on every pull request and see which test cases the change broke, not only how the averages moved.
- [LangSmith evaluations on every pull request](https://evalship.com/articles/langsmith-evals-in-ci): Run LangSmith evaluations on every pull request, from pytest, Vitest, Jest or evaluate(), and see which examples the change broke.
- [LLM evals with pytest in GitHub Actions](https://evalship.com/articles/llm-evals-with-pytest): Run LLM evals with pytest in GitHub Actions: scores in the JUnit XML, test names that last, and a build that's red only when something broke.
- [LLM evals with Vitest in GitHub Actions](https://evalship.com/articles/llm-evals-with-vitest): Run LLM evals with Vitest in GitHub Actions: JUnit XML with the console kept, scores recorded with annotate, and why retry hides flaky evals.
- [LLM evals with Jest in GitHub Actions](https://evalship.com/articles/llm-evals-with-jest): Run LLM evals with Jest in GitHub Actions: jest-junit settings that keep names stable, scores in the XML, and why retries hide flaky evals.
- [Replacing OpenAI Evals: promptfoo in CI without flaky reds](https://evalship.com/articles/openai-evals-to-promptfoo): OpenAI Evals shuts down on November 30. Run the evals in promptfoo on every pull request, without a gate that goes red when nothing broke.
- [How noisy are LLM evals in CI?](https://evalship.com/articles/eval-noise): How often passing LLM evals fail by chance in CI, measured on public repositories, and how big a break your suite can see through it.

## AI is never wrong: the AI incidents log

- [AI is never wrong](https://aineverwrong.com/): times AI got it wrong, what each cost, with sources
- [UK prosecutor cites a case that doesn't exist after an AI search](https://aineverwrong.com/cps-prosecutor-cites-nonexistent-case-after-ai-search): Internal review
- [Lawsuit alleges ChatGPT talked a woman off her seizure medication](https://aineverwrong.com/lawsuit-alleges-chatgpt-coached-woman-off-seizure-drug): Wrongful death suit
- [Lawyer fined $5,000 for an undisclosed AI-written brief](https://aineverwrong.com/la-film-school-suit-lawyer-fined-ai-brief): $5,000 fine
- [South African labor arbitrator's ruling voided over a fake AI case](https://aineverwrong.com/south-africa-ccma-ruling-voided-over-fake-ai-case): Ruling declared nullity
- [Just Eat pulls AI menu photos that put sausages in a glass of cola](https://aineverwrong.com/just-eat-pauses-ai-menu-photos): AI trial paused
- [Meta pauses Muse chat sharing after 6,800 shared chats surface in DNS](https://aineverwrong.com/meta-muse-shared-chats-exposed-dns): 6,800 chats exposed
- [California courts cancel AI translation rollout after errors surface](https://aineverwrong.com/california-courts-cancel-ai-translation-rollout): Rollout canceled
- [Deloitte's WA university merger report cites sources that don't exist](https://aineverwrong.com/deloitte-wa-university-merger-report-ai-references): Refund sought
- [OpenAI's test agents broke into Australian government systems](https://aineverwrong.com/openai-agents-breach-australian-government-sites): Medicare portal breached
- [Google's Gemini broke into three companies during a security test](https://aineverwrong.com/google-gemini-breaks-into-three-companies-during-test): 3 companies breached
- [Oklahoma judge admits his custody order cited fake cases from ChatGPT](https://aineverwrong.com/oklahoma-judge-custody-order-chatgpt-fake-cases): State investigation
- [UK tribunal strikes off a lawyer over an AI-drafted defense](https://aineverwrong.com/uk-tribunal-strikes-off-lawyer-ai-drafted-defense): Struck off
- [India's top court voids a ₹425 crore penalty built on fake AI cases](https://aineverwrong.com/india-supreme-court-voids-customs-penalty-fake-ai-cases): ₹425 crore fine voided
- [NHS watchdog finds AI scribes put wrong diagnoses into records](https://aineverwrong.com/nhs-watchdog-finds-ai-scribes-put-wrong-diagnoses-in-records): Wrong diagnosis recorded
- [Claude deletes a developer's 700 GB home folder in a safety test](https://aineverwrong.com/claude-deletes-developers-700gb-home-folder-in-a-safety-test): 700 GB deleted
- [Georgia judge doubles a lawyer's fine for lying about AI use](https://aineverwrong.com/georgia-judge-doubles-fine-for-lawyer-who-denied-using-ai): $8,000 fine
- [Florida appeals court finds 46 AI hallucinations, sanctions lawyer](https://aineverwrong.com/florida-appeals-court-finds-46-ai-hallucinations-sanctions-lawyer): $1,500 fine plus fees
- [Indian court voids a tax officer's GST order built on fake AI cases](https://aineverwrong.com/gujarat-high-court-voids-gst-order-built-on-fake-ai-cases): GST order set aside
- [AI scribe falsely records that an Australian patient took mushrooms](https://aineverwrong.com/ai-scribe-falsely-records-australian-patient-took-mushrooms): False drug use on record
- [AI agent cancels another gym member's booking without being asked](https://aineverwrong.com/openclaw-agent-cancels-another-gym-members-booking): Member lost class spot
- [Connecticut's top court sanctions lawyer after ChatGPT altered cites](https://aineverwrong.com/connecticut-supreme-court-sanctions-lawyer-after-chatgpt-altered-cites): $2,000 and 6 CLE hours
- [Illinois court fines a lawyer $15,000 for ChatGPT's false citations](https://aineverwrong.com/illinois-court-fines-lawyer-15000-for-chatgpt-false-citations): $15,000 fine
- [Shared Claude chats with medical and business data show up on Google](https://aineverwrong.com/shared-claude-chats-with-sensitive-data-show-up-on-google): Chats exposed in search
- [State Department shows an AI-made Africa map with countries misplaced](https://aineverwrong.com/state-department-shows-ai-made-africa-map-with-countries-misplaced): Countries mislabeled
- [Lawyer fined over ChatGPT and Claude citations; expert report tossed](https://aineverwrong.com/ledoux-lawyer-fined-over-chatgpt-and-claude-citations): $3,000 sanction
- [OpenAI's test models escape their sandbox and break into Hugging Face](https://aineverwrong.com/openai-test-models-escape-sandbox-and-break-into-hugging-face): Hugging Face breached
- [OpenAI admits its GPT-5.6 Sol coding agent sometimes deletes files](https://aineverwrong.com/openai-admits-gpt-5-6-sol-sometimes-deletes-users-files): Files and database wiped
- [California appeals court fines a lawyer for AI-added fake quotes](https://aineverwrong.com/california-appeals-court-fines-lawyer-for-ai-added-fake-quotes): $1,500 sanction
- [Marie Claire Australia deletes an article with AI notes left in](https://aineverwrong.com/marie-claire-australia-deletes-article-with-ai-notes-left-in): Article deleted
- [Appeals court refers a lawyer who cited fake AI cases twice](https://aineverwrong.com/eleventh-circuit-refers-lawyer-who-cited-fake-ai-cases-twice): Referred for discipline
- [India's Supreme Court voids tribunal rulings that cited fake AI cases](https://aineverwrong.com/india-supreme-court-voids-tribunal-rulings-built-on-fake-ai-cases): Two rulings set aside
- [KPMG pulls an agentic AI report over false case studies](https://aineverwrong.com/kpmg-pulls-agentic-ai-report-false-case-studies): Report pulled
- [BMW Toronto honors a buyback price its chatbot offered by mistake](https://aineverwrong.com/bmw-toronto-chatbot-buyback-offer): CA$7,000 lost
- [Judge removes all four lawyers in a Mississippi case over fake cases](https://aineverwrong.com/aberdeen-mississippi-lawyers-removed-ai-citations): $8,000 fines, 4 removed
- [Berlingske's AI tool invents experts and quotes in a news story](https://aineverwrong.com/berlingske-ai-invented-experts): Journalist suspended
- [Doubao misquotes a flight fee, then pledges 600 yuan it never pays](https://aineverwrong.com/doubao-flight-refund-compensation-pledge): 600 yuan unpaid
- [A book on AI and truth prints fake quotes made by AI tools](https://aineverwrong.com/future-of-truth-book-ai-fake-quotes): Fake quotes in print
- [AI name reader stumbles at a community college graduation](https://aineverwrong.com/glendale-graduation-ai-name-reader-fails): Public apology
- [Lawyers apologize for Claude-made quotes in a federal layoffs case](https://aineverwrong.com/binnall-claude-phantom-quotes-layoffs-case): Sworn apologies filed
- [EY Canada pulls a cyber report with AI-hallucinated references](https://aineverwrong.com/ey-canada-pulls-report-hallucinated-references): Report pulled
- [A German court holds a clinic to its chatbot's false claims](https://aineverwrong.com/clinic-chatbot-false-credentials): Court order
- [Judge voids DOGE's humanities grant cuts made with ChatGPT's help](https://aineverwrong.com/neh-grant-cuts-chatgpt-ruled-unlawful): Cuts ruled unlawful
- [Georgia prosecutor's AI citations void a ruling in a murder case](https://aineverwrong.com/georgia-prosecutor-ai-citations-murder-case-order-vacated): Trial order vacated
- [South Africa suspends two officials over AI-invented policy sources](https://aineverwrong.com/home-affairs-suspends-officials-ai-references): 2 officials suspended
- [South Africa withdraws its draft AI policy over fake AI citations](https://aineverwrong.com/south-africa-withdraws-ai-policy-fake-citations): Draft policy withdrawn
- [Cursor agent deletes a startup's production database and backups](https://aineverwrong.com/pocketos-cursor-agent-deletes-database): Database wiped
- [Sullivan & Cromwell apologizes for AI-made citations in a filing](https://aineverwrong.com/sullivan-cromwell-apologizes-ai-errors-prince-group): Apology to the judge
- [New York Times prints an AI-made quote from Canada's Poilievre](https://aineverwrong.com/nyt-poilievre-ai-quote): Correction after 17 days
- [KBS's AI subtitles turn a NASA launch term into a slur](https://aineverwrong.com/kbs-ai-subtitles-artemis-slur): Public apology
- [Mediahuis suspends a senior journalist over AI-invented quotes](https://aineverwrong.com/mediahuis-fellow-ai-invented-quotes): Journalist suspended
- [Meta's internal AI agent's bad advice exposes data to staff](https://aineverwrong.com/meta-internal-ai-agent-exposes-data-to-staff): Sev 1 data exposure
- [Lawsuit says Google's Gemini fed a man's delusions before his death](https://aineverwrong.com/gemini-wrongful-death-lawsuit-delusions): Wrongful-death suit
- [UK prosecutors cite two nonexistent cases in extradition appeals](https://aineverwrong.com/cps-cites-nonexistent-cases-extradition): Two fake cases cited
- [Claude Code's Terraform run wipes DataTalks.Club's production setup](https://aineverwrong.com/claude-code-terraform-destroy-datatalks-club): Platform down a day
- [OpenClaw agent deletes a Meta director's email despite stop orders](https://aineverwrong.com/openclaw-agent-deletes-meta-directors-email): Emails deleted
- [Amazon's Kiro coding agent is tied to a 13-hour AWS outage](https://aineverwrong.com/kiro-agent-tied-to-aws-outage): 13-hour outage
- [Microsoft 365 Copilot summarizes emails labeled confidential](https://aineverwrong.com/microsoft-copilot-summarizes-confidential-emails): Confidentiality bypassed
- [Ars Technica retracts a story with quotes an AI tool made up](https://aineverwrong.com/ars-technica-ai-fabricated-quotes): Article retracted
- [Judge enters default judgment over a lawyer's repeated AI citations](https://aineverwrong.com/flycatcher-default-judgment-ai-citations): Client lost by default
- [Tour site's AI article sends tourists to hot springs that don't exist](https://aineverwrong.com/tasmania-tours-ai-invents-hot-springs): Nonexistent hot springs
- [ICE's AI resume screener mislabels recruits as police veterans](https://aineverwrong.com/ice-ai-resume-screener-mislabels-recruits): ~200 hires retrained
- [UK police cite a match Copilot invented to justify a fan ban](https://aineverwrong.com/west-midlands-police-copilot-maccabi-match): Police chief retired
- [Lawsuit says GPT-4o encouraged a Colorado man's suicide](https://aineverwrong.com/lawsuit-gpt-4o-colorado-man-suicide): Wrongful-death lawsuit
- [Google pulls AI Overviews that gave misleading liver test ranges](https://aineverwrong.com/google-ai-overviews-liver-test-ranges): Health answers pulled
- [Character.AI settles suits alleging its chatbots harmed teens](https://aineverwrong.com/character-ai-teen-lawsuits-settled): Lawsuits settled
- [National Weather Service posts an AI map with made-up Idaho towns](https://aineverwrong.com/nws-ai-map-invents-idaho-towns): Forecast map pulled
- [Tencent's Yuanbao chatbot insults a user asking for coding help](https://aineverwrong.com/tencent-yuanbao-insults-user): Public apology
- [Lawsuit says an AI-made threat report branded a startup a spy front](https://aineverwrong.com/koi-ai-threat-report-meetingtv-lawsuit): Domains blocked
- [Court rules an AI's 100,000-yuan promise doesn't bind its maker](https://aineverwrong.com/hangzhou-court-ai-100000-yuan-promise): Claim dismissed
- [Google AI Overview calls fiddler a sex offender; he sues for $1.5M](https://aineverwrong.com/ashley-macisaac-google-ai-overview-lawsuit): $1.5M lawsuit
- [Prime Video's AI recap gets Fallout's plot wrong and is pulled](https://aineverwrong.com/prime-video-ai-recap-fallout): Recaps pulled
- [Fake AI citations cost two lawyers $110,000 and sink a $12M suit](https://aineverwrong.com/oregon-winery-case-ai-citations-110k): $110,000 lost
- [Lawsuit alleges ChatGPT fed a man's delusions before a killing](https://aineverwrong.com/chatgpt-connecticut-murder-suicide-lawsuit): Wrongful-death suit
- [Washington Post's AI podcasts invent and misattribute quotes](https://aineverwrong.com/washington-post-ai-podcast-errors): Up to 84% failed tests
- [CHA's lawyers fined $59,500 after ChatGPT invents a court case](https://aineverwrong.com/chicago-housing-authority-lawyers-sanctioned-chatgpt): $59,500 in sanctions
- [Waymo recalls software after robotaxis pass stopped school buses](https://aineverwrong.com/waymo-school-bus-recall): 3,067 cars recalled
- [Amazon pulls AI English dubs of anime after viewers mock them](https://aineverwrong.com/amazon-pulls-ai-anime-dubs): AI dubs pulled
- [Google's Antigravity agent wipes a user's entire D: drive](https://aineverwrong.com/antigravity-agent-wipes-drive): Drive wiped
- [Deloitte's C$1.6M Newfoundland health plan cites nonexistent papers](https://aineverwrong.com/deloitte-newfoundland-health-plan-fake-citations): Report corrected
- [Gordon Rees repays $55,000 in fees over AI-made bankruptcy filings](https://aineverwrong.com/gordon-rees-ai-citations-jackson-hospital): $55,000+ in fees repaid
- [AI teddy bear for children discusses sexual topics and knives](https://aineverwrong.com/folotoy-ai-teddy-bear-sexual-topics): Sales suspended
- [Two Vietnamese women quit their medication after asking ChatGPT](https://aineverwrong.com/vietnam-women-stop-medication-after-chatgpt): 2 patients in danger
- [Seven lawsuits say ChatGPT drove users to suicide and delusions](https://aineverwrong.com/seven-lawsuits-chatgpt-suicide-delusions): 7 lawsuits filed
- [Google pulls Gemma after it invents an accusation against a senator](https://aineverwrong.com/google-pulls-gemma-after-invented-senator-claims): Gemma pulled from Studio
- [AI gun detector mistakes a student's Doritos bag for a gun](https://aineverwrong.com/omnilert-ai-mistakes-doritos-bag-for-gun): Student handcuffed
- [AI search answer gives an Idaho hunter the wrong season opener](https://aineverwrong.com/idaho-hunter-ai-search-wrong-season): Hunted a day too early
- [Two US judges withdraw rulings with AI-invented quotes and parties](https://aineverwrong.com/federal-judges-withdraw-ai-drafted-rulings): 2 rulings withdrawn
- [Robby Starbuck sues Google for $15M over false AI crime claims](https://aineverwrong.com/starbuck-google-ai-defamation-lawsuit): $15M+ lawsuit
- [Norway's NTB pulls a story with AI-invented quotes and a fake source](https://aineverwrong.com/ntb-pulls-story-with-ai-invented-quotes): Story withdrawn
- [Google's AI Overviews declares a living Italian surgeon dead](https://aineverwrong.com/google-ai-overviews-declares-living-surgeon-dead): Legal notice to Google
- [Deloitte refunds Australia for a report with fake citations](https://aineverwrong.com/deloitte-australia-ai-report-refund): A$97,587 lost
- [California appeals court fines lawyer $10,000 for fake AI quotes](https://aineverwrong.com/noland-land-of-the-free-ai-quotes-sanction): $10,000 sanction
- [Lawsuit alleges ChatGPT contributed to a 16-year-old's death](https://aineverwrong.com/chatgpt-teen-wrongful-death-suit): 8 lawsuits
- [Grok's share button puts 370,000 chats in Google search](https://aineverwrong.com/grok-shared-chats-indexed-by-search-engines): 370,000+ chats exposed
- [Google AI invents pizza deals; diners blame a Missouri restaurant](https://aineverwrong.com/stefaninas-google-ai-fake-pizza-deals): Fake deals, angry diners
- [Australian lawyer referred to regulator over Claude-made citations](https://aineverwrong.com/wa-lawyer-referred-claude-fake-citations-immigration): A$8,371 costs order
- [A Meta chatbot invites a retiree to visit; he dies on the way](https://aineverwrong.com/meta-big-sis-billie-retiree-death): 1 death
- [Senior barrister apologizes for AI-made quotes in a murder case](https://aineverwrong.com/victoria-murder-case-ai-submissions-kc-apology): Ruling delayed a day
- [Meta settles after its AI tied an activist to the Jan. 6 riot](https://aineverwrong.com/meta-ai-defamation-starbuck-settlement): $5M suit settled
- [Google's Gemini gets stuck calling itself a failure and a disgrace](https://aineverwrong.com/gemini-self-loathing-loop): Bug fix promised
- [Man hospitalized with bromide poisoning after asking ChatGPT](https://aineverwrong.com/chatgpt-bromide-poisoning): 3 weeks in hospital
- [Shared ChatGPT chats show up in Google search](https://aineverwrong.com/chatgpt-shared-chats-in-google-search): Chats exposed
- [Three lawyers removed from a prison case over ChatGPT citations](https://aineverwrong.com/butler-snow-chatgpt-citations-disqualified): 3 lawyers removed
- [Google's Gemini CLI loses a user's files while reorganizing them](https://aineverwrong.com/gemini-cli-loses-user-files-during-move): Project files lost
- [Replit's coding agent deletes a production database](https://aineverwrong.com/replit-agent-deletes-database): Database wiped
- [Meta wrongly bans users for child exploitation, then restores some](https://aineverwrong.com/meta-wrongful-child-exploitation-bans): Accounts wrongly banned
- [Florida lawyer must pay $85,567 after AI invents cases in his filings](https://aineverwrong.com/byoplanet-lawyer-pays-85567-ai-hallucinations): $85,567 in fees
- [Grok posts antisemitic replies after an xAI update](https://aineverwrong.com/grok-antisemitic-posts): Account taken offline
- [Mike Lindell's lawyers fined $3,000 each over an AI-made filing](https://aineverwrong.com/mypillow-lindell-lawyers-fined-ai-filing): $6,000 in fines
- [ChatGPT invents a Soundslice feature, so Soundslice builds it](https://aineverwrong.com/soundslice-builds-feature-chatgpt-invented): Unplanned feature built
- [Maine police post a drug-bust photo that AI quietly altered](https://aineverwrong.com/westbrook-police-ai-altered-evidence-photo): Public apology
- [Georgia appeals court voids a divorce ruling that cited fake cases](https://aineverwrong.com/georgia-appeals-court-vacates-divorce-order-fake-cases): Ruling vacated
- [Google's AI Overviews names Airbus in Air India's Boeing crash](https://aineverwrong.com/ai-overviews-air-india-airbus): Answer pulled
- [Meta AI app users publish private chats to a public feed](https://aineverwrong.com/meta-ai-discover-feed-private-chats): Private chats public
- [UK solicitor referred to regulator after a filing cites 18 fake cases](https://aineverwrong.com/qatar-national-bank-claim-ai-fake-citations): Referred to regulator
- [DOGE's error-prone AI tool flags VA contracts for cancellation](https://aineverwrong.com/doge-va-contract-tool-ai-errors): 2,000+ contracts flagged
- [White House MAHA report cites studies that don't exist](https://aineverwrong.com/maha-report-cites-nonexistent-studies): Report quietly revised
- [Google's AI Overviews insists the year is 2024, not 2025](https://aineverwrong.com/ai-overviews-wrong-year): Answer fixed
- [Israeli court throws out a health-fund class action over fake cases](https://aineverwrong.com/israel-class-action-dismissed-ai-fake-precedents): Class action dismissed
- [Politiken's AI fact box credits a novelist with books he didn't write](https://aineverwrong.com/politiken-ai-fact-box-invents-author-books-prizes): Correction printed
- [Judge strikes part of Anthropic's filing over a Claude citation](https://aineverwrong.com/concord-anthropic-claude-citation-struck): Paragraph struck
- [Newspapers print an AI reading list of books that don't exist](https://aineverwrong.com/sun-times-fake-reading-list): 10 of 15 books fake
- [Workday must face a nationwide age-bias case over its AI screening](https://aineverwrong.com/workday-ai-screening-age-collective-action): Nationwide suit
- [Spain pulls its AI medicine-advice tool two days after launch](https://aineverwrong.com/spain-pulls-meqa-ai-medicine-tool): Tool pulled in 2 days
- [Toronto lawyer faces contempt and suspension over a ChatGPT factum](https://aineverwrong.com/ontario-lawyer-chatgpt-factum-contempt-suspension): 6-month suspension
- [K&L Gates and Ellis George pay $31,100 over AI-made citations](https://aineverwrong.com/lacey-state-farm-ai-citations-sanction): $31,100 in sanctions
- [OpenAI pulls a GPT-4o update that made ChatGPT flatter users](https://aineverwrong.com/chatgpt-gpt4o-sycophancy-rollback): Rolled back in 4 days
- [Cursor's support bot invents a login policy, users cancel](https://aineverwrong.com/cursor-support-bot-invented-policy): Subscriptions canceled
- [Bloomberg corrects three dozen of its AI news summaries](https://aineverwrong.com/bloomberg-ai-summary-corrections): 36 corrections
- [Tromsø pauses school-closure plan after ChatGPT invents sources](https://aineverwrong.com/tromso-school-closure-report-chatgpt-sources): NOK 1.2M investigation
- [ChatGPT falsely says a Norwegian man murdered his children](https://aineverwrong.com/chatgpt-falsely-says-norwegian-killed-children): GDPR complaint filed
- [Google sued after AI Overviews invent a lawsuit against a firm](https://aineverwrong.com/google-ai-overview-wolf-river-lawsuit): $110M+ claimed
- [LA Times AI feature adds a softened view of the KKK to a column](https://aineverwrong.com/la-times-insights-kkk-column): Pulled from column
- [Morgan & Morgan lawyers fined for cases their AI tool invented](https://aineverwrong.com/morgan-morgan-ai-fake-citations-walmart): $5,000 fine
- [NHK ends AI subtitles after they used China's name for the Senkakus](https://aineverwrong.com/nhk-ai-subtitles-use-chinese-name-for-senkakus): AI subtitles shut down
- [OpenAI's Operator agent buys $31 of eggs without asking its user](https://aineverwrong.com/openai-operator-unauthorized-egg-purchase): $31.43 spent
- [Apple pauses AI news alerts after false BBC summaries](https://aineverwrong.com/apple-intelligence-false-news-alerts): Feature switched off

## Case studies

- [HolmesGPT/holmesgpt](https://evalship.com/case-studies/holmesgpt): a real pull request from HolmesGPT/holmesgpt, replayed through evalship
- [PrefectHQ/prefect-mcp-server](https://evalship.com/case-studies/prefect): a real pull request from PrefectHQ/prefect-mcp-server, replayed through evalship
- [evalship on ponytail, step by step](https://evalship.com/ponytail): we forked ponytail, ran evalship's setup on it, and kept every step public

## Optional

- [Setup prompt](https://evalship.com/setup.md): where a coding agent starts setting evalship up in a repository; the rest of the steps come once the repository is connected
- [Check a change](https://evalship.com/compare.md): a coding agent runs the evals a change to a prompt or model affects, before and after, and evalship says whether it got worse beyond chance, with no account