Skip to content

For AI agents: this site's index is at /llms.txt, and each docs page comes as Markdown at its URL plus .md.

Case study · nommer, our own app

Dutch recipes stopped falling back to English

nommer is our recipe app. It translates each recipe into Dutch, German and Japanese with one AI call per language. In November 2026 the Dutch call came back as cut-off JSON every time, so Dutch readers got English. Our coding agent tried newer models with Optimize. evalship said which one won, and when it couldn't tell yet.

20/30
replies the old model got right in 10 tries. The Dutch one failed every time.
30/30
replies flash-lite-3.1 got right, beyond chance
−46%
waiting for each translation
$0.0025
per recipe translated, up from $0.0011

What Optimize said

  1. Nov 3 · 3 tries

    Three tries can't tell

    The agent tried two newer Gemini models, flash-lite-3.1 and flash-3.8, against google/gemini-2.5-flash-lite, the one nommer ran. Each got 3 tries on the 54 strings of nommer's translation eval. Both got 9 of 9 replies right, against 6 of 9. Chance gives a gap this size about 1 time in 10. So evalship picked neither, and said what would settle it: 8 more tries per test case, about $0.17.

    The agent took only flash-lite-3.1 further: flash-3.8 did as well at 16 times the cost.

    evalship.com/gh/gglucass/nommer-app/optimize

    dca Model

    Newer model for translation drafts

    Nothing beat current yet
    Question
    Can recipe translations (nl, de, ja) be drafted by a newer Gemini model so they stop failing and falling back to English?
    Why this
    Every translation draft is served by the fallback google/gemini-2.5-flash-lite (backend/app/models/translation.rb:19, :165) because the Cerebras-pinned gemma call always 404s (translation.rb:150-153), and translation-eval passes only 18 of 54 on main, with the whole nl batch failing on cut-off JSON.
    Answer
    Nothing beat current yet: no change passed more often beyond chance, but more tries could still tell.
    Next

    8 more tries per test case of current, flash-lite-3.1, and flash-3.8, about $0.17, would settle them.

    currentgoogle/gemini-2.5-flash-lite

    Passed 6 / 9 replies

    Cost per run $0.0011

    Time per run 220.8 s

    flash-lite-3.1Within chance

    Between 10 points less often and 69 more often. Chance gives a gap this size about 1 time in 10.

    Passed 9 / 9 replies

    Cost per run +132%

    Time per run −47%

    flash-3.8Within chance

    Between 10 points less often and 69 more often. Chance gives a gap this size about 1 time in 10.

    Passed 9 / 9 replies

    Cost per run +1538%

    Time per run +195%

    How evalship tested it

    Each variant ran the same 54 test cases, 3 tries each, from commit bc72a85. evalship pools the replies and counts a gap only when chance gives it less than 1 time in 80, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.

    flash-lite-3.1 google/gemini-3.1-flash-lite instead of google/gemini-2.5-flash-lite as the translation draft model (backend/app/models/translation.rb:19)

    Passed more often (18)

    • nl: Preheat the oven to 190°C/gas mark 53/3, current 0/3
    • nl: Cover the beetroot with foil and place in the oven until tender, approximately 30–45 minutes3/3, current 0/3
    • nl: Cook for 6 minutes3/3, current 0/3
    • nl: Turn and continue to cook for a further 4 minutes3/3, current 0/3
    • nl: Finish in the oven for 2 minutes at 220°C/gas mark 63/3, current 0/3
    • nl: Remove from the oven and allow to rest for 5–7 minutes3/3, current 0/3
    • nl: After 1 minute, add the maple syrup and coat the duck until it reduces down to a sticky glaze3/3, current 0/3
    • nl: Cut each chicken into 4 pieces and reserve the backbones for stock3/3, current 0/3
    • nl: Prep 1/4 cup flat-leaf parsley: mince3/3, current 0/3
    • nl: Cut the slab bacon into 1- by 1/4- by 1/4-inch sticks3/3, current 0/3

    and 8 more

    flash-3.8 google/gemini-3.8-flash instead of google/gemini-2.5-flash-lite as the translation draft model (backend/app/models/translation.rb:19)

    Passed more often (18)

    • nl: Preheat the oven to 190°C/gas mark 53/3, current 0/3
    • nl: Cover the beetroot with foil and place in the oven until tender, approximately 30–45 minutes3/3, current 0/3
    • nl: Cook for 6 minutes3/3, current 0/3
    • nl: Turn and continue to cook for a further 4 minutes3/3, current 0/3
    • nl: Finish in the oven for 2 minutes at 220°C/gas mark 63/3, current 0/3
    • nl: Remove from the oven and allow to rest for 5–7 minutes3/3, current 0/3
    • nl: After 1 minute, add the maple syrup and coat the duck until it reduces down to a sticky glaze3/3, current 0/3
    • nl: Cut each chicken into 4 pieces and reserve the backbones for stock3/3, current 0/3
    • nl: Prep 1/4 cup flat-leaf parsley: mince3/3, current 0/3
    • nl: Cut the slab bacon into 1- by 1/4- by 1/4-inch sticks3/3, current 0/3

    and 8 more

  2. Nov 3 · 10 tries

    Ten tries can

    10 fresh tries of flash-lite-3.1: 30 of 30 replies, against 20 of 30. Chance gives a gap this size less than 1 time in 40. So evalship picked it, with what it means: 132% more per run, 46% less waiting.

    evalship.com/gh/gglucass/nommer-app/optimize

    4fd Model

    Gemini 3.1 Flash Lite for translation drafts, 10 tries

    flash-lite-3.1 passes more often Shipped
    Question
    Does drafting recipe translations (nl, de, ja) on Gemini 3.1 Flash Lite stop the batches that fail and fall back to English?
    Why this
    The 3-try run (evalship-optimize/7) had flash-lite-3.1 at 9 of 9 replies against current's 6 of 9, within chance; current's draft model google/gemini-2.5-flash-lite (backend/app/models/translation.rb:19) cuts off the nl batch's JSON on every try.
    Answer
    flash-lite-3.1 passes more often, beyond chance: 30 of 30 replies, against 20 of 30 for current. Chance gives a gap this size less than 1 time in 40.
    What it means

    For you: 132% more expensive per run: $0.0025 instead of $0.0011.

    For your users: 46% less waiting, and results that pass more often.

    Next

    Shipped: #10 merged.

    currentgoogle/gemini-2.5-flash-lite

    Passed 20 / 30 replies

    Cost per run $0.0011

    Time per run 220.8 s

    flash-lite-3.1Better

    15 to 52 points more often. Chance gives a gap this size less than 1 time in 40.

    Passed 30 / 30 replies

    Cost per run +132%

    Time per run −46%

    How evalship tested it

    Each variant ran the same 54 test cases, 10 tries each, from commit bc72a85. evalship pools the replies and counts a gap only when chance gives it less than 1 time in 40, so a round of many variants can't win by luck. Under each variant, the range its gap could be, with no gap at the solid line. A gap matters at 5 points of pass rate, the dashed lines: a clearly cheaper change that passes at most that much less often is as good, and one that can't gain that much isn't worth more tries.

    flash-lite-3.1 google/gemini-3.1-flash-lite instead of google/gemini-2.5-flash-lite as the translation draft model (backend/app/models/translation.rb:19)

    Passed more often (18)

    • nl: Preheat the oven to 190°C/gas mark 510/10, current 0/10
    • nl: Cover the beetroot with foil and place in the oven until tender, approximately 30–45 minutes10/10, current 0/10
    • nl: Cook for 6 minutes10/10, current 0/10
    • nl: Turn and continue to cook for a further 4 minutes10/10, current 0/10
    • nl: Finish in the oven for 2 minutes at 220°C/gas mark 610/10, current 0/10
    • nl: Remove from the oven and allow to rest for 5–7 minutes10/10, current 0/10
    • nl: After 1 minute, add the maple syrup and coat the duck until it reduces down to a sticky glaze10/10, current 0/10
    • nl: Cut each chicken into 4 pieces and reserve the backbones for stock10/10, current 0/10
    • nl: Prep 1/4 cup flat-leaf parsley: mince10/10, current 0/10
    • nl: Cut the slab bacon into 1- by 1/4- by 1/4-inch sticks10/10, current 0/10

    and 8 more

  3. Nov 3 · 23 minutes later

    Shipped the same day

    Our coding agent opened #10, a one-line change of the draft model, with evalship's verdict in its description. It merged 23 minutes after the three-try answer. The agent also noted that the call had no retry after a rate limit, which became #11.

How we made this

  • nommer is our own app. Its repo is private, so this page names its pull requests without linking them.
  • The tries ran on Nov 3, 2026, on our machine. Claude Code (Opus 5.5) followed evalship's optimize prompt (v3) on nommer's commit bc72a85 and ran nommer's translation eval through OpenRouter. Both experiments' AI calls cost about 10 cents.
  • The three-try experiment is as evalship stored it, plus the model current ran, which the prompt didn't ask for yet. The ten-try one is rebuilt from the agent's session log, because our test server's database has been reset since. The log has every try's result and evalship's answer at upload. It doesn't have each try's cost, tokens or time, so those come from the three-try experiment, matched to the percentages evalship reported that day.
  • The answers come from evalship's code as it is today, and both verdicts are the ones it gave on Nov 3. Back then, the three-try answer didn't yet say how many more tries would settle it: the agent chose ten itself.

See what evalship finds in your evals

Ask your coding agent: Claude Code Codex Cursor GitHub Copilot

Set up https://evalship.com/setup.md

More case studies: HolmesGPT/holmesgpt · PrefectHQ/prefect-mcp-server