Evaluation, observability and guardrails

Look at your data first. Then measure what you found.

Most AI evaluation starts with a dashboard of generic scores and ends with nobody trusting them. We start by reading your real traces with someone from your side who knows the domain, name the ways the system actually fails, and only then build the checks. The result is an evaluation suite you own, that runs before every change and grows from production.

See it in our work

At a glance

Starts with
About a hundred traces, read
Your time
One domain expert, a few hours a week in month one
Verdicts
Pass or fail, with a written reason
Existing systems
First findings usually within a week

What it covers

Measure the ways your system actually fails.

  • Error analysis before metrics

    We read real traces, a hundred or so to start, and write in plain words what went wrong in each. Those notes are grouped into a small set of named failure modes and counted. The most frequent failure is the first thing we fix, and the first thing we measure.

  • Three gulfs, one diagnosis

    Every failure sits in one of three gaps: we did not understand the data, we did not tell the model what we meant, or the model did not generalise from what we told it. Naming the gulf tells you the fix: look at more data, tighten the specification, or change the architecture, the retrieval or the model.

  • Evaluators that earn their trust

    Where a check can be code, it is code: is the citation there, does the number match the database, is the format right. Where it needs judgement, a model judges against a written rubric, and we validate that judge against human labels before it is allowed to grade anything. Every check is pass or fail with a reason, never a score out of five.

  • A labelled set you own

    The traces your team marked, with the reason, become the set every change runs against. Part of it is held back so nobody tunes to the test. It grows from production, and it is yours: plain data, in your account, readable by anyone.

How we do it

Read the traces, then write the checks.

  1. 01

    One domain expert, final say

    Someone from your side who knows what a right answer looks like reads the traces with us and marks them. One person with the last word, not a committee, so the labels stay consistent and the work moves.

  2. 02

    Read, then code

    Open notes on each trace first, in the reviewer’s own words. Then the notes are grouped into named failure modes and counted. A taxonomy with numbers beside it, not a list of worries.

  3. 03

    Diagnose by gulf, fix the biggest first

    Each failure mode gets a gulf and a fix. Specification failures are fixed in the prompt and the spec. Generalisation failures need decomposition, better retrieval or a different model. Comprehension failures send us back to the data. The most frequent one goes first.

  4. 04

    Build the checks, validate the judges

    Assertions in code for everything that can be checked mechanically. A judge with a rubric for the rest, measured against the human labels until it agrees often enough to be trusted, and re measured when it changes.

  5. 05

    Wire it in, keep reading

    The suite runs on every change and blocks the release when it fails. In production, traces are sampled and read on a cadence, corrections feed the labelled set, and the taxonomy is updated when a new failure appears. Evaluation is a loop, not a launch task.

What you get

A suite your team keeps running.

  • A failure taxonomy for your system, with counts
  • A labelled trace set your team marked, with a held out portion
  • Code checks and validated judges, with their agreement against human labels
  • A regression gate in your deployment pipeline
  • Tracing, cost and latency in your account, and a review cadence your team keeps
  • A model swap procedure: run the suite, show the diff, decide

Where we have done it

Real engagements, named.

Questions

Questions we get asked.

Can we not just use an evaluation dashboard?
Generic scores like helpfulness or coherence measure nothing about the ways your system fails. They look reassuring and they move for reasons nobody can explain. The checks that matter are the ones that came out of reading your own traces, which is why we start there.
Why pass or fail, and not a score out of five?
A four out of five tells you nothing you can act on, and two reviewers will give different fours. Pass or fail with a written reason is something a person can agree with, a judge can be validated against, and a pipeline can block on.
How many traces do we need to read?
About a hundred to start. New failure modes stop appearing well before that in most systems, and the counts tell you where to spend the first fix. The set keeps growing from production after launch.
Who labels, and how much of their time?
One person from your side who knows the domain, with the final say. A few hours a week during the first month, then a lighter cadence. Their marks are the ground truth; our judges are measured against them, not the other way round.
What about retrieval, tools and long conversations?
Each is measured where it fails. Retrieval is checked on its own: did the right document come back before the model wrote anything. Tool use is checked on the call and its arguments. A conversation is judged at the turn where it went wrong, not on the whole transcript.
We already have an AI system someone else built. Can you add this?
Yes, and it is a common starting point. Error analysis on your real traffic tells you what the system actually does today, usually within a week, and the suite is built from that.

What clients say

In their words.

01 / 04

Life Insurance Direct
Before Metricsense, checking calls was a manual process: we selected files for review. Now the platform analyses every call, and we review by risk. It has identified customer experiences we would not have picked up before, and it lets us scale our quality assurance program without increasing headcount, with our team still the human in the loop. It helps us protect our customers, our business and our agents.

Russell Cain

Founder and CEO, Life Insurance Direct

Also under AI engineering

Where it leads next.

All of AI engineering

Not sure this is the right place to start?