Shadow traffic: a practical guide

Mirror real requests to a new agent, score everything, change nothing. The safest way to find out what a release will do. At Signa we spend most of our time reading traces, and most of what we know comes from failures our customers were brave enough to share.

Where it starts

Most teams begin with a spreadsheet of fifty questions and a thumbs-up column. It is a fine start and a poor finish: it measures the model, not the agent, and it never sees the long tail.

What we do instead

We score the whole run — retrieval, tool choice, arguments, final answer and policy — and we weight each case by how often it happens and how much it costs when it goes wrong.

Every automated judge is calibrated against human graders before its scores count toward a release gate.

What to take away

Measure the thing your customers experience, calibrate the thing that measures it, and keep the evidence. The rest is tooling.