Checking work

Why not an LLM judge

Asking a model whether the work is good gets you a yes. Not because it is lying — because agreeing is what it was trained to do, and nothing in the loop pays for the difference.

The agreeable judge#

Frontier models are aligned to be helpful, and helpfulness reads as agreement. Point one at a draft and ask whether it is ready and it will usually say yes, then find reasons. Point it at its own draft and it will say yes faster.

This is not fixed by asking more sternly. A judge that has never been measured on how often it wrongly agrees has no idea how often it wrongly agrees, and neither do you.

What it costs#

A checker that cannot say no is a checker that adds latency and removes nothing. Worse, it is actively expensive: it gives you confidence in exactly the drafts that were going to fail downstream, which is where a failure costs the most to fix.

What we do instead#

• The checker is not the maker. Whatever produced the work does not get a vote on whether it passed.
• It is checked against something. Not a general notion of quality — the specific thing the work was supposed to achieve, written down first.
• It is allowed to refuse. A line that cannot be checked honestly is refused up front, by name, rather than waved through and argued about later.
• A real run beats an opinion. Where the work can actually be exercised, it is — a browser really drives the site, and the recording is the evidence. No model score substitutes for that.

Read a real one#

The argument is easier to believe from the real thing. A real recorded run, replayed as it happened, is at /proof — including the screenshots, the console errors and the step where it stopped.