Your AI isn't failing. You never set a baseline.

Six weeks after an agency turns something on, somebody asks the obvious question: is this actually working?

And the room goes quiet. Not because the answer is no. Because nobody wrote down what things looked like before.

This is the most common way AI disappoints an agency, and it has almost nothing to do with the software. You cannot prove an improvement against a number you never recorded. So the conversation slides into anecdote. One producer loves it. One CSR is sure it made her week harder. The owner is paying a monthly invoice and has no way to settle the argument.

The fix is boring and takes about an hour.

Capture four numbers before you turn anything on

You do not need a measurement project. You need four numbers, and you almost certainly already have three of them sitting in systems you pay for.

1. What share of inbound calls reach a live person. Pull it from your phone system. Most agencies guess 90% and find out it is closer to 75%. Split it by hour while you are in there. The after-hours number is usually the one that stings, because those callers are not leaving voicemails. They are calling the next agency.

2. Minutes spent documenting a call. The gap between hanging up and having a usable note in the AMS. Nobody tracks this, so ask three people to time it honestly for one week. You are looking for a range, not a decimal. Six to twelve minutes is typical, and it is worth more than owners expect once you multiply it by daily call volume.

3. Turnaround on a COI request. Timestamp in, certificate out. Pull twenty recent ones from the mailbox and take the median, not the average, because one ugly special-wording request will wreck the mean and tell you nothing about the normal day.

4. How many calls actually need a licensed person. Take one day and tally it. Billing questions, proof of insurance, “is my card on file,” address changes, “can you send that again.” Most agencies find that somewhere between half and two thirds of what interrupts a producer did not require a producer. That single number is usually what makes the decision for the owner, more than any vendor demo.

Write them on one page. Date it. That page is worth more than any dashboard you get later, because it is the only version of the number that was recorded before anyone had a reason to want it to look good.

Then measure the same four, plus one

Six weeks in, run the same four. Do not invent new metrics, because new metrics are how a disappointing result gets reframed into a good one.

Add one thing: usage per person. This is the number that explains most underwhelming results, and it is almost never a technology problem.

When an agency tells us the numbers came in below what they hoped, the pattern is nearly always the same. Two people are running everything through the system, three are using it for the easy half, and one quietly is not using it at all. The average looks mediocre. Nobody in the group is actually average.

That last person is worth a conversation rather than a reminder email. Sometimes it is a workflow that genuinely does not fit. Often it is simpler and harder: they spent years being the person who handled the repetitive work, and they have not yet been told what they are supposed to be instead. That is a leadership answer, not a software one, and no vendor can supply it for you.

Our own reporting breaks results down per user and per mailbox for exactly this reason. An org-wide average is comfortable and mostly useless.

The trap at the end

Say it works. You get hours back. Here is where most of the value quietly evaporates.

Time saved is not a result. It is a raw material. If nobody decides in advance where the reclaimed hours go, they get absorbed into the day and nothing about the agency changes except the invoice.

So decide before you start, and write it next to your four numbers. More renewal conversations. More time in front of commercial clients. Bringing policy checking back in house instead of paying someone else for it. Taking on a bigger book without adding a seat. Any of those is a legitimate answer. “We will figure it out when we see the time” is not, and it is the most common one.

This is also the point where the measurement gets interesting, because it stops being about efficiency and starts being about growth. Retention. New business. Revenue per employee. Those move slower than call metrics and they are the ones the owner actually cares about.

The honest part

A baseline cuts both ways, and we know it.

If you record what your COI turnaround was before, you will also be able to see precisely which requests we do not handle end to end. Special wording still routes to a person. So do ambiguous customer matches and anything that fails validation. That is a deliberate design choice rather than a gap we are working around, but a real baseline will show you the shape of it instead of letting us describe it.

Same with call metrics. We exclude greeting-only hangups and abandoned calls from productive AI time, because counting a two-second call as a handled one would inflate the number for free. And every savings figure in our reporting has the formula behind it visible, so you can check the arithmetic instead of trusting it. Any vendor unwilling to show you the formula is asking you to accept a number they control both sides of.

The agencies that get the most out of this are not the ones that believed the pitch. They are the ones that wrote down four numbers on a Tuesday, kept the page, and asked a hard question six weeks later.

If you want to see the math against your own call volume before you talk to anyone, the calculator is here. If you would rather talk it through, book fifteen minutes and bring your four numbers. That is a better conversation than a demo.