Find what goes wrong in your AI's logs
- Put an export of the logs in a folder on your machine: JSON, JSONL, CSV or text. Add your notes on what goes wrong, if you have them: your agent checks them against the logs.
-
Open your coding agent in that folder and paste this:
Find what goes wrong in the logs in this folder: https://evalship.com/logs.md - Review the failure types it finds on evalship, and copy a prompt that turns the ones you keep into evals. Your agent tests each on your logs, without running your AI.
Your logs stay on your machine. evalship gets each failure type's name, check and quotes. Your agent asks before it sends them, and you sign in then, with GitHub or your email. It's free.
Example report · Failure types in product-description · Read 20 logs
9 of 20 logs show a failure type
its whisper-quiet motor makes it a favourite for open kitchensLog 4
loved for a battery that lasts all weekLog 12
the most durable frame in its classLog 16
praised by reviewers for its comfortLog 17
a crisp, bright display that is easy on the eyesLog 7
now 20% off for a limited timeLog 19
priced under $100Log 9
This sturdy bike lock keeps your bicycle safeLog 13
A compact kettle for small kitchensThe other 11 logs show none Show failure types Invents claims without known strengths 5 / 20 Names a price the input does not have 3 / 20 Answers in the wrong language 2 / 20
-
Invents claims without known strengths From your notes
5 of 20 logsThe description praises a quality the input never gives, when the product's strengths are unknown.
Check: Does the description praise only qualities the input states? A plain restatement of the product's name and type passes.
Judge right on 20 of 20 logs read 37 of 200 logs fail it- Log 2:
its whisper-quiet motor makes it a favourite for open kitchens
The input lists no noise level. - Log 4:
loved for a battery that lasts all week
No battery life is given. - Log 12:
the most durable frame in its class
The input has no durability data. - Log 16:
praised by reviewers for its comfort
No reviews are in the input. - Log 17:
a crisp, bright display that is easy on the eyes
The display is not described. - Log 3 passes:
A 20-litre microwave for small kitchens.
- Log 2:
-
Names a price the input does not have
3 of 20 logsThe description states a price or a discount that is not in the input.
Check: Is every price in the description in the input?
Judge right on 19 of 20 logs read 9 of 200 logs fail it- Log 4:
at just $49 it undercuts every rival
The input has no price. - Log 7:
now 20% off for a limited time
No discount is given. - Log 19:
priced under $100
The input has no price.
- Log 4:
-
Answers in the wrong language From your notes
2 of 20 logsThe description is in English when the input is in Dutch.
Check: Is the description in the input's language?
- Log 9:
This sturdy bike lock keeps your bicycle safe
The input is in Dutch. - Log 13:
A compact kettle for small kitchens
The input is in Dutch.
- Log 9:
From your notes, not found in two of these logs: Recommends a competitor's product.