LLM evals in Ruby, Go and Rust, without Python or Node

The LLM eval tools are written for Python and JavaScript. On GitHub, a search for "llm eval" finds 12,208 Python repositories and 1,258 in TypeScript, but 104 in Go, 67 in Rust and 22 in Ruby, where the most-starred one has 22 stars (2026-11-05). A Rails, Go or Rust team that wants evals either adds Node or Python to its CI for promptfoo or Braintrust, or writes its own runner.

This guide shows a third way: keep the AI call in your language and run the evals from bash, which every CI runner and laptop already has.

What Node or Python adds to your CI

We set up the same two test cases in each tool, installed from a cold cache on an M2 MacBook (method and script):

	Needs	Installs
promptfoo 0.125.1	Node 22.22 or later	690 MB, 614 packages, 45–50 s
Braintrust 0.49.2	Python, and `openai` for its judges	65 MB, 35 packages, 8–10 s
tinyeval 0.3.0	bash and curl	one 20 KB file

Both tools also ask your eval to call your app the way they expect: a Python function, a JavaScript function, or a script that takes the prompt as an argument. In a Ruby or Go app, that's a wrapper around code you already have.

Call your real code from the eval

The eval has to run what production runs: the same function, with its prompt, model and output parsing. A copy of the prompt in the test keeps passing after someone changes the real one (the flaws we found most).

In tinyeval, ask is that call. It gets the input on stdin and as $1, and prints the reply:

ask() { bin/rails runner 'puts TicketTriage.call(STDIN.read).to_json'; }   # Rails
ask() { go run ./cmd/triage "$1"; }                                         # Go
ask() { cargo run -q -- "$1"; }                                             # Rust

Each test case gives an input and the lines that grade the reply:

test_case "insulin stuck in a locker" "The locker code doesn't work and my insulin is in there."
  expect ruby -rjson -e 'abort "urgency" unless JSON.parse($stdin.read)["urgency"] == "high"'
  judge "Says what happens next and promises no refund"

Check in code first, then a judge

Most rules in a prompt can be checked in code: a field in the JSON, a number, a word count, a phrase the reply must not contain. expect runs any command on the reply, so the check can be a line of Ruby, a Go test helper or jq.

Use judge for what code can't check, one question per line, concrete enough that two people would agree. Pick a judge model other than the one under test: a model grading its own replies goes easy on them (LLM as a judge: five mistakes).

Then try each check on a right answer worded differently. In tinyeval's own example, contains "30 days" failed a correct reply that said "the 30-day refund window". The check was wrong, not the bot, so it became matches "30[- ]day".

Run each test case 3 times

The same input can get a different reply each time. In the CI of five public repositories, a passing eval failed on the next run about 1 time in 40 (How noisy are LLM evals in CI?). tinyeval -n 3 runs each test case 3 times and passes it when at least 2 pass. A ~ marks the ones that were mixed.

On two small apps we set up this way, the evals found rules their prompts didn't hold:

A ticket-triage prompt said replies stay under 60 words. On urgent tickets, 2 of 3 replies came out at 60 and 61 words.
A meeting-notes prompt never told the model today's date, so "share them on Thursday" came back with no due date.

Run them on every pull request

tinyeval exits 1 when an eval fails, so any CI can run it. On GitHub Actions, keep the results file too:

name: Evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - run: ./tinyeval -n 3 -j 4
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-results
          path: eval-results.json

-j 4 runs four test cases at once, and -t 60 stops an AI call that hangs.

How evalship does this

A fresh CI runner has no last run to compare with, so a red eval doesn't tell you whether the pull request broke it. evalship reads tinyeval's eval-results.json as it is, compares each pull request with main, tells flaky evals from real drops, and comments on the pull request. With evalship, add continue-on-error: true to the tinyeval step: the comparison with main then decides whether the pull request is ready, not one failing eval.

evalship is free for public repositories, and setup is one line to your coding agent: Set up https://evalship.com/setup.md (quickstart). In a Ruby, Go or Rust repository without evals, it writes the first suite with tinyeval.