24 articles in AI engineering
AI evaluation is how you find out whether your AI system is right, on your data, before a customer does. This is the whole picture: what an eval is, how it differs from a benchmark, what to measure, how to build your first set, and where it sits in the pipeline.
When an AI assistant answers wrongly, the first instinct is to rewrite the instructions. Half the time the instructions are fine and the model never saw the figure at all. Here is how retrieval works, the failure that looks like a model failure, how to tell them apart, and the funds firm where the returns turned out to be pictures.
Our data agent got worse the longer a conversation ran. No prompt change fixed it. When we profiled the context window, stale tool output was eating most of it. We replaced each old tool result with a one line placeholder and cut input tokens by 38% on the first turn and 37.6% across a 25 step run.
Ask an AI model for a figure it does not have and it will give you one anyway, fluently, and wrong. If your name goes on the numbers, that is not a risk you can prompt away. Here is the pattern we use so an assistant cannot invent a figure, explained from the start, and the climate data publisher it runs for today.
AI generates answers in seconds. Confidence is not accuracy. Three questions that separate effective AI users from ones who amplify mistakes.
Everyone agrees you should "look at your data." Then they open a hundred traces, scroll for ten minutes, feel vaguely worried, and reach for a tool. The looking has a method — open coding, then axial coding into a ranked taxonomy — and the taxonomy, not your assumptions, is what decides which evals to write.
Most teams build one corner of the eval map — offline, end-to-end, on the inputs they imagined — and call it "evals," the way you'd say "we have tests." Eval is plural. Here's the map: Hamel Husain's three levels, and the three questions that decide which kind of eval you're actually writing.
The first eval post said pull a hundred production traces and read them. But what if you haven't launched? Here's how to build your first eval set from dimensions, scenarios, and synthetic data — and why that set is scaffolding, not the building.
Most teams buy an eval tool, run it once, and call it done. They confuse benchmarks with evals — and ship AI that confidently produces wrong outputs no one will catch. The work the tool can't do for you is the work that matters: looking at your traces, naming the failure modes, and writing assertions that fire when they recur.