Recorded LLM replies in Rails tests pass any prompt change

Many Rails apps test their AI calls by replaying a recorded API response with VCR or WebMock. We read the tests of 39 open-source Ruby apps that call an LLM (2026-11-06). 19 of them replay the model's HTTP response. In 16 of those 19, the replay matches without looking at the request, so a test passes whatever prompt or model the code sends.

What the tests do with the AI call	Apps
Replay a recorded response, matching any request	16
Replay it only when the request body matches	3
Stub the client object, so no HTTP at all	14
No test of the AI call found	4

An app and its two forks share their tests, so they count once.

A cassette from 2025 still passes

One personal-finance app, with over 10,000 stars, categorizes transactions with OpenAI. Its test replays a cassette recorded in April 2025. The recorded request used gpt-4.1-mini and a prompt saying a category and a transaction's classification should match.

Today the code sends gpt-4.1 and the opposite rule: any category can go with any transaction, income or expense. On 6 November its CI ran 14,711 tests, and the categorizer test passed. It would also pass with an empty prompt.

Why the replay ignores your prompt

VCR finds a recorded response by the request's method and URL, unless you tell it more. Every call to OpenAI's Responses API is a POST to the same URL, so every recording matches every prompt. A WebMock stub_request without .with works the same way.

VCR.configure do |c|
  c.cassette_library_dir = "test/vcr_cassettes"
  c.hook_into :webmock
  # no match_requests_on: VCR matches on [:method, :uri]
end

The default is reasonable for most APIs. A request to fetch a user is the same request each time, and its body is noise: timestamps, ids, multipart boundaries. An LLM request is different, because the body is the prompt, the model and the settings: the behavior you care about.

What these tests still check

The replays aren't useless. They check that the app builds a request, parses the reply and handles errors. Keep them for that. They just don't tell you whether a prompt or model change still gives right answers, and a green build reads as if they do.

Make a prompt change fail the replay

Match LLM cassettes on the body as well, and never record on CI:

VCR.configure do |c|
  c.default_cassette_options = {
    match_requests_on: [:method, :uri, :body],
    record: ENV["CI"] ? :none : :once
  }
end

Now a changed prompt or model raises VCR::Errors::UnhandledHTTPRequestError until someone records the cassette again, with a real key. 3 of the 19 apps do this. It tells you the request changed, which is worth knowing, but not whether the new reply is still right. And a cassette re-recorded once records whatever the model said that day.

Stubs that assert on the request body have the same limit, plus one more: a few apps build the expected body from the same constant the code uses, so a prompt edit changes both sides and still passes.

Test the reply, not the recording

To know whether a prompt change still works, run the real call on fixed inputs and check the reply: in code where you can (a category from the list, a JSON field, a word count), with a model as judge where you can't. Run each input more than once, because the same input can get a different reply. LLM evals in Ruby, Go and Rust shows how, from bash, without adding Python or Node to CI.

5 of the 39 apps had evals like that. The finance app is one: it has golden datasets and an eval runner, but its CI runs them only on release tags, so a pull request that changes the prompt sees only the replay.

How evalship does this

evalship's setup reads your repository's tests and evals and lists the ones that can't fail, with the file and line. A replay that ignores the request was the first finding in the review of a Rails app with 54,000 stars. evalship then compares each pull request's eval results with main, and says which test cases the change broke.

Setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart).

Method

We searched GitHub code on 2026-11-06 for Ruby apps that call OpenAI, Anthropic or Gemini, through ruby-openai, ruby_llm, the Anthropic SDK or plain HTTP, and that record or stub those calls in tests. For each app we read the VCR or WebMock setup and the tests of its AI calls at the default branch's latest commit. We left out 58 SDKs and client libraries, which send no prompts of their own.

The search favors apps that use VCR and OpenAI, so the share may differ in apps that don't. Apps that stub the client object pass a prompt change too, unless the test checks the prompt. Three apps put the model in the URL (Bedrock, Gemini), so a model change fails their replays, but a prompt change doesn't.