Evaluation is the product

There is a particular conversation that happens about eight weeks into an enterprise AI pilot. Somebody senior asks a version of the question: is this thing actually good? And the room goes quiet, because the honest answer is that nobody knows. There is a demo that works. There is a spreadsheet where someone graded forty outputs in an afternoon three sprints ago. There is a general feeling that it seems better than last month.

That gap, between having a system and being able to say anything defensible about it, is where most enterprise AI projects quietly die. Not in a dramatic failure. They just never accumulate enough evidence to justify the next decision, and eventually the budget goes somewhere with a clearer story.

The measurement problem is the actual problem

It is worth being precise about why this happens, because the usual diagnosis is wrong. Teams do not skip evaluation because they are lazy or unsophisticated. They skip it because building a real evaluation set is harder than building the feature, and it is harder in a way that is unglamorous and unrewarded.

To evaluate a clinical coding assistant you have to decide what a correct code is, in cases where two qualified coders disagree. To evaluate a contract analysis tool you have to find the clauses where the answer depends on a definition three documents away. To evaluate a support agent you have to enumerate the situations in which it should refuse to answer at all. Each of these requires domain expertise, sustained attention, and a tolerance for argument. None of them demo well.

So the work gets deferred. And once deferred, it becomes structurally very hard to reintroduce, because by then the system is in front of users and every hour spent on measurement is an hour not spent on the bug someone is complaining about today.

What the literature already tells us

The uncomfortable part is that the failure modes are well documented and teams still walk into them.

Take citation. A grounded system that quotes its sources feels trustworthy, that is precisely the point of showing citations. But a 2023 Stanford study of four widely used generative search engines found that, on average, only 51.5% of generated sentences were fully supported by their citations, and only 74.5% of citations actually supported the sentence they were attached to (Liu, Zhang & Liang, 2023). These were production systems from serious organisations. The citations were right there on screen, and roughly half the claims were not properly backed by them.

Nobody sets out to ship that. It happens because "has citations" is a feature you can see and "citations are faithful" is a property you have to measure.

Or take the now-standard practice of using a strong model as a grader. LLM-as-a-judge works: on MT-Bench, GPT-4's agreement with human preferences exceeded 80%, which is roughly the level at which humans agree with each other (Zheng et al., 2023). That is a genuinely useful result. But the same line of work documents position bias, verbosity bias, and self-preference, a judge model tends to favour the first option it sees, longer answers, and its own outputs. If you deploy an unvalidated judge you have not solved measurement; you have added a second unmeasured system to the stack and put it in charge of grading the first.

The pattern in both cases is the same. The mechanism is real and useful. The failure only becomes visible when you point a measurement at it.

What we do instead

We now treat the evaluation suite as the first deliverable of an engagement, before any architecture is settled. In practice that means four things.

Define correct with the people who will have to defend it. Not the project sponsor, the clinician, the underwriter, the compliance officer who owns the outcome. This conversation is slow and it surfaces disagreements that the organisation has been quietly carrying for years. That is a feature. A specification that survives it is worth something.

Collect the hard cases first. A random sample of production traffic is close to useless as an evaluation set, because the interesting behaviour lives in the tail. We want the ambiguous documents, the ones with conflicting sources, the ones where a competent human would ask a clarifying question. A suite of 200 genuinely hard cases beats 5,000 easy ones, and it fits in CI.

Validate the grader before trusting it. If a model is grading outputs, the grader itself gets a test set with known human labels, and we report its agreement rate. Where agreement is weak, that dimension goes to human review and we say so, rather than quietly reporting a number we do not believe. Position and verbosity bias get controlled for by randomising order and checking whether length predicts score.

Put it in the repository and in CI. The suite lives next to the code, runs on every change, and produces a report with pass/fail gates. This is the part that converts evaluation from a project into a property of the system. If it only runs when someone remembers, it will stop running by the second month.

What it costs

We should be honest about the trade. Building the harness first typically adds two to four weeks before anything user-visible exists, at the exact moment when a project is most under pressure to show progress. That is a real cost and it is genuinely uncomfortable for everyone involved.

What you buy is the ability to make every subsequent decision on evidence: to know that switching models improved things rather than assuming it, to expand scope where the numbers support it, to walk into a governance review with a document instead of an anecdote. Over a nine-month engagement the harness pays for itself several times over, mostly by preventing work that would have been wasted.

It also produces the thing enterprise buyers actually need and rarely get: an artefact they can run themselves, against their own data, after we leave.

Why we open-source it

Our evaluation harness is going out under a permissive licence, and so are the benchmark definitions built on top of it. This is not altruism with a good story attached, it is the most efficient thing we can do.

An assertion that we are careful about measurement is worth approximately nothing on its own, every engineering organisation claims that. A harness a sceptical engineer can clone and run against their own corpus is a different kind of claim, because it can be checked rather than believed.

That is roughly the whole strategy: build the measurement layer, publish it, and let people verify the claim rather than take it.

---

If you are running an AI system that has to survive a governance review and you are not sure what you would show, we are happy to talk it through, including if the conclusion is that you do not need us. Write to engineering@rudvanth.com.