Measure Before Installing an AI Chatbot: A 14-Day Baseline

Before installing an AI chatbot, spend 14 days recording how your current enquiry path performs: enquiries received, how many were qualified, which details were missing, how long the first human response took and how many became bookings. Without that baseline, almost any result after launch is ambiguous, and you cannot tell improvement from coincidence.

A scenario#

A plumbing business adds a website chatbot in late spring. A month later, enquiries are up. The owner is pleased, until someone points out that it was the first hot month of the year, a Google Ads campaign started the same week and the website was redesigned in between.

Did the chatbot help? Nobody can say. A two-week baseline taken before launch, with notes on what else was happening, would have given the owner something to compare against.

The six measures#

Record these from your existing channels for 14 days:

1. Website sessions on priority service pages, such as hot water, blocked drains and emergency callouts. Use the analytics tool you already have.
2. Contact-form submissions and tracked calls. If you do not track calls, a tally sheet by the phone is better than nothing.
3. Qualified enquiries, using a written definition (see below).
4. Missing-information rate: the share of enquiries with no suburb, no service type or no usable contact detail.
5. Time to first human response, split into staffed hours and after hours.
6. Booked jobs or appointments that came from those enquiries.

A simple spreadsheet works: one row per enquiry, one column per measure.

Write the definition of "qualified" first#

Decide before you count. For example, a qualified enquiry has:

• valid contact details;
• a location inside your service area;
• a service you actually offer;
• a stated need or problem.

Use the same definition for the baseline and after launch. Do not redefine it after seeing the numbers; that is how measurement turns into storytelling.

Record the context#

Next to the numbers, keep a short log:

• Was a Google Ads or social campaign running?
• Was there a storm, heatwave, school holiday or public holiday?
• Did the website change?
• Was a staff member away, slowing responses?
• Did prices or services change?

A baseline is only useful if its context is preserved. Two weeks during a storm surge are not comparable with two quiet weeks.

After launch: keep the old paths open#

Leave your phone number and contact form in place when the chatbot goes live. Then compare lead quality by channel:

• Chat may increase volume but lower fit, for example more out-of-area enquiries.
• Or it may produce fewer but richer enquiries, with suburb and job type already filled in.

Both are plausible. The baseline lets you see which one is happening.

Small numbers need patience#

A low-traffic site might get eight enquiries in two weeks. Do not claim the chatbot "doubled leads" because the next fortnight had sixteen. Report raw counts with caveats and extend the test period before drawing conclusions.

Higher-traffic sites can do better: show the chatbot on some service pages and not others, or alternate weeks, and compare.

Running the pilot with Pivra#

Be clear about what each stage measures:

• Free plan: website chat with 500 messages a month and one chatbot, no card required (pricing). It is useful for checking answer quality on your own content during the baseline period, before any visitor sees it. It does not include lead capture or analytics, so it cannot measure enquiries.
• Paid plans: lead capture, analytics and handover start at Starter. Any paid plan can be tried free for seven days, with a card needed at checkout.

A seven-day trial is shorter than a 14-day baseline. Treat the trial as a check that the workflow works end to end (fields captured, alerts arriving, handover reaching a person), not as the measurement period. For a fair before-and-after comparison, plan for at least 14 days on a paid plan, or accept that a shorter window gives only directional evidence.

Once live, Pivra's analytics show conversations, leads captured and a lead conversion rate over the last 30 days (analytics), and you can export leads as CSV to line them up against your baseline spreadsheet (lead capture). For setup steps, see how it works.

To turn the comparison into a value estimate, use the method in AI chatbot ROI for small business.

Set a decision date#

Write three things down before launch:

• The decision date, for example 30 days after go-live.
• The criteria: for instance, qualified enquiries at least equal to baseline, missing-information rate lower, no critical wrong answers.
• The options: continue, change the workflow, or remove the chatbot.

Removing it is a legitimate outcome. A pilot that ends with "this didn't help us" has still done its job.

Baseline checklist#

• Definition of "qualified" written and shared.
• Spreadsheet with one row per enquiry.
• Call tracking or a manual tally in place.
• Context log started.
• Response times recorded for staffed and after-hours periods.
• Decision date and criteria agreed.

Next step: spend 14 days measuring the current path before calling the new one better.