Look at your data first. Then measure what you found.
Most AI evaluation starts with a dashboard of generic scores and ends with nobody trusting them. We start by reading your real traces with someone from your side who knows the domain, name the ways the system actually fails, and only then build the checks. The result is an evaluation suite you own, that runs before every change and grows from production.
What it covers
Error analysis before metrics
We read real traces, a hundred or so to start, and write in plain words what went wrong in each. Those notes are grouped into a small set of named failure modes and counted. The most frequent failure is the first thing we fix, and the first thing we measure.
Three gulfs, one diagnosis
Every failure sits in one of three gaps: we did not understand the data, we did not tell the model what we meant, or the model did not generalise from what we told it. Naming the gulf tells you the fix: look at more data, tighten the specification, or change the architecture, the retrieval or the model.
Evaluators that earn their trust
Where a check can be code, it is code: is the citation there, does the number match the database, is the format right. Where it needs judgement, a model judges against a written rubric, and we validate that judge against human labels before it is allowed to grade anything. Every check is pass or fail with a reason, never a score out of five.
A labelled set you own
The traces your team marked, with the reason, become the set every change runs against. Part of it is held back so nobody tunes to the test. It grows from production, and it is yours: plain data, in your account, readable by anyone.
How we do it
One domain expert, final say
Someone from your side who knows what a right answer looks like reads the traces with us and marks them. One person with the last word, not a committee, so the labels stay consistent and the work moves.
Read, then code
Open notes on each trace first, in the reviewer’s own words. Then the notes are grouped into named failure modes and counted. A taxonomy with numbers beside it, not a list of worries.
Diagnose by gulf, fix the biggest first
Each failure mode gets a gulf and a fix. Specification failures are fixed in the prompt and the spec. Generalisation failures need decomposition, better retrieval or a different model. Comprehension failures send us back to the data. The most frequent one goes first.
Build the checks, validate the judges
Assertions in code for everything that can be checked mechanically. A judge with a rubric for the rest, measured against the human labels until it agrees often enough to be trusted, and re measured when it changes.
Wire it in, keep reading
The suite runs on every change and blocks the release when it fails. In production, traces are sampled and read on a cadence, corrections feed the labelled set, and the taxonomy is updated when a new failure appears. Evaluation is a loop, not a launch task.
- 1
One domain expert, final say
Someone from your side who knows what a right answer looks like reads the traces with us and marks them. One person with the last word, not a committee, so the labels stay consistent and the work moves.
- 2
Read, then code
Open notes on each trace first, in the reviewer’s own words. Then the notes are grouped into named failure modes and counted. A taxonomy with numbers beside it, not a list of worries.
- 3
Diagnose by gulf, fix the biggest first
Each failure mode gets a gulf and a fix. Specification failures are fixed in the prompt and the spec. Generalisation failures need decomposition, better retrieval or a different model. Comprehension failures send us back to the data. The most frequent one goes first.
- 4
Build the checks, validate the judges
Assertions in code for everything that can be checked mechanically. A judge with a rubric for the rest, measured against the human labels until it agrees often enough to be trusted, and re measured when it changes.
- 5
Wire it in, keep reading
The suite runs on every change and blocks the release when it fails. In production, traces are sampled and read on a cadence, corrections feed the labelled set, and the taxonomy is updated when a new failure appears. Evaluation is a loop, not a launch task.
What you get
- A failure taxonomy for your system, with counts
- A labelled trace set your team marked, with a held out portion
- Code checks and validated judges, with their agreement against human labels
- A regression gate in your deployment pipeline
- Tracing, cost and latency in your account, and a review cadence your team keeps
- A model swap procedure: run the suite, show the diff, decide
Where we have done it
India Avenue: 50 real visitor questions, each marked right or wrong with the reason by the firm. That set became the suite every change must pass, and it is why they can take a newer model the week it lands.
India AvenueLife Insurance Direct: their QA lead reviewed calls beside the system every week and told us where it was wrong, with the reason. We tuned against real disagreements, and every new check is one plain sentence that runs on every call from then on.
Life Insurance DirectOnlyFacts: four mechanical checks before anything ships. Does the number match the database, is it the right chart, is the citation there, has a banned word crept back in.
OnlyFacts
Questions we get asked
Can we not just use an evaluation dashboard?
Generic scores like helpfulness or coherence measure nothing about the ways your system fails. They look reassuring and they move for reasons nobody can explain. The checks that matter are the ones that came out of reading your own traces, which is why we start there.
Why pass or fail, and not a score out of five?
How many traces do we need to read?
Who labels, and how much of their time?
What about retrieval, tools and long conversations?
We already have an AI system someone else built. Can you add this?
Also under ai engineering
Talk it through
Tell us what you need from evaluation, observability and guardrails.
Thirty minutes with one of our architects. We will tell you whether it is a pilot, a build, or not worth doing yet.