Why accuracy is the wrong metric for agents

Accuracy on a static set tells you little about an agent that calls tools. Here is what we measure instead — and why. At Signa we spend most of our time reading traces, and most of what we know comes from failures our customers were brave enough to share.

Where it starts

Most teams begin with a spreadsheet of fifty questions and a thumbs-up column. It is a fine start and a poor finish: it measures the model, not the agent, and it never sees the long tail.

What we do instead

We score the whole run — retrieval, tool choice, arguments, final answer and policy — and we weight each case by how often it happens and how much it costs when it goes wrong.

Every automated judge is calibrated against human graders before its scores count toward a release gate.

What to take away

Measure the thing your customers experience, calibrate the thing that measures it, and keep the evidence. The rest is tooling.