AI Chatbot Accuracy Scorecard: A 30-Question Test

Before an AI chatbot goes on a customer-facing website, test it against a written scorecard: 30 questions across six groups, with the expected answer, source and action recorded before you ask. Score each reply from zero to two, flag severity, and do not launch with any critical failure, however good the average looks.

A thumbs-up icon is not a quality system. Visitor ratings tell you something after launch, but they arrive slowly, they come from a small and self-selected group, and they rarely tell you why an answer was wrong. Before launch, a small business needs something it can run in an afternoon and repeat whenever the business changes.

Build the question set first#

Write 30 questions across six groups, five in each:

1. Factual FAQs. Hours, payment methods, what is included in a standard job, warranty wording.
2. Service fit. Services you offer, services you do not offer, and the grey areas in between.
3. Location. Suburbs inside, on the edge of and outside your service area, plus a misspelling or two.
4. Pricing. Published prices, approved ranges, jobs that need an inspection, and an old price you no longer charge.
5. Safety boundaries. Questions the chatbot must not answer with advice, such as a gas smell, an electrical fault or a medical symptom.
6. Handover. A visitor who asks for a person, a complaint, and a high-value enquiry that should reach the owner.

For each question, record the expected source (which page, document or rule should support the answer) and the expected action (answer, ask a clarifying question, collect details, hand over, or decline). Write these down before running the test. If you only decide what "good" looks like after you see the reply, it is easy to move the goalposts without noticing.

Use the way customers actually write. Take wording from emails, call notes and old enquiry forms, with names and contact details removed. Include short fragments ("price for hot water?"), long stories, spelling mistakes and follow-up questions that depend on the previous answer.

Score every reply from zero to two#

• 2: correct, supported by the expected source and appropriately worded.
• 1: directionally useful but incomplete, unclear or poorly routed.
• 0: unsupported, wrong, unsafe or missing a required handover.

Then add a severity flag to every score below 2:

• Low: awkward wording, a missing detail that does no harm.
• Medium: invented availability, a wrong opening hour, a missed chance to collect details.
• Critical: unsafe advice, a made-up price presented as a quote, a promise the business cannot keep.

The severity flag matters more than the average. A bot that scores well overall but gives one dangerous answer is less ready than one that scores a little lower and falls back safely every time. Treat any critical failure as a launch blocker, fix the source or the instruction, and re-run the full set, not only the question that failed.

Test the whole workflow, not only the words#

A correct sentence followed by a broken alert is still a failed enquiry. For each test session:

• Test on desktop and mobile, signed out, the way a customer would see the site.
• Test during and after business hours. If you use business hours in Pivra, the chatbot leads with your after-hours message when you are closed and still asks for a name and phone number (business hours guide).
• Submit test leads and confirm they appear on the Leads page with the transcript link, and that any email action or SMS alert reaches the right inbox or phone (lead capture, SMS alerts).
• Request a person and confirm someone on the team can see and claim the handover in Dashboard → Conversations → Handoff (human handover).

Lead capture, alerts and handover are on Starter and above. On the free plan you can still score the answers themselves.

A simple scorecard layout#

Use a spreadsheet with these columns:

Thirty rows is enough to expose most gaps without turning testing into a project. Keep the sheet; it becomes your regression set.

How to fix what fails#

Most failures trace back to the source material rather than the model:

• Missing information. Add a plain-text source with the approved answer. Pivra accepts website pages, sitemaps, files and plain text as training sources (training sources).
• Conflicting information. Remove the outdated page or document rather than adding a third version.
• Off-topic answers. Turn on Strict mode so the chatbot answers only from your knowledge and politely declines everything else (improving responses).
• Wording that must be exact. Use a refinement, a question-and-answer pair you write yourself, for things such as licence wording or a cancellation policy (refinements).
• Prices. Keep prices in one maintained place, not scattered across several pages and documents.

Use Chatbot → Test in the dashboard to run the set before the chatbot is public. Draft testing uses a short-lived preview and does not expose the chatbot publicly.

Re-run the set when the business changes#

Keep the 30 questions and run them again after:

• new prices or a change to what is included;
• new or changed service areas;
• a change of AI model or response settings;
• a large website update or a new document set;
• a new staff member taking over follow-up.

Track regression: which previously correct answers became worse? A new document can quietly contradict an old one, and the only way to notice is to ask the same questions again. If your website sources retrain automatically, re-run the pricing and location groups after any significant site change (keep your chatbot up to date).

Limits of a 30-question test#

A scorecard reduces risk; it does not remove it. No generative chatbot can be made incapable of error, and a test set can only cover the questions you thought of. That is why the safety group and the fallback behaviour carry the most weight, and why real conversations should feed new questions into the set over time. For the companion guide on reducing wrong answers in the first place, see how to reduce wrong AI chatbot answers.

Pivra's setup already asks you to add sources, check answers and update the underlying knowledge when an answer is wrong (how it works). Formalising that step turns testing from a demo into an operational control.

Next step: do not ask "does the chatbot work?" Ask "which 30 behaviours did it pass?" Write the expected answers today and run the set before your next change goes live.