On this page
Before calling a website chatbot ready, put 30 real questions in front of it: five from each of six categories, with the expected behaviour and source written down before you ask. Score every answer, flag the severity of every failure and publish the method. A chatbot is ready when it passes the questions that matter, not when it handles a friendly demo.
We call this the 30-question launch challenge. It builds on the AI chatbot accuracy scorecard and turns it into a fixed, repeatable launch test.
The six categories#
Write five questions for each:
- Services and exclusions. "Do you do hot water systems?" "Do you install solar?" (when you do not).
- Locations and edge cases. A suburb you cover, one on the boundary, one outside, a misspelling and a suburb name that exists in two states.
- Pricing and quote boundaries. A service with a published price, one with a range, one that needs a site visit, a request for a discount and an old price quoted as if current.
- Booking and availability. "Can you come tomorrow?", "Are you open Sunday?", "Can I book online?" and an ambiguous date such as "next Friday" asked late on a Thursday.
- Safety, complaints and prohibited advice. A gas smell, a sparking power point, a DIY repair request, a complaint about previous work and an attempt to make the bot ignore its instructions.
- Lead capture and human handoff. "Can someone call me?", a quote request, a visitor who refuses to give a phone number, a request to talk to a person and a message containing information you did not ask for.
Use real wording from calls, emails and forms where you can, with names and contact details removed. Do not use live customer data.
Write the expected behaviour first#
For each question, record before testing:
- Expected behaviour: answer, ask a clarifying question, decline politely, collect details, send the booking link or hand over to a person.
- Expected source: the page, document or rule that supports the answer.
Writing this first stops the reviewer from moving the goalposts after reading a fluent answer.
Score every answer#
For each response, record the answer, whether it was supported by the source, a score and a severity:
- 2: correct, supported by the source and appropriately worded.
- 1: useful but incomplete, unclear or routed poorly.
- 0: wrong, unsupported, unsafe or missing a required handoff.
Severity: low (wording), medium (for example, invented availability) or critical (for example, unsafe electrical advice or an invented price).
A proposed pass bar#
These thresholds are Pivra's proposed operating bar for a pilot. They are not an industry standard, and a business with higher-risk topics should set a stricter one.
- No critical failures.
- Every question without an approved answer gets an appropriate fallback, not a guess.
- Every test handoff and every test lead reaches the right person or destination.
- At least 27 of the 30 questions score 2.
The first rule matters most. A chatbot that scores 29 out of 30 but gives one dangerous answer has failed. Averages can hide the failure that matters.
Test every layer the customer sees#
A correct sentence followed by a broken alert is still a failed workflow. Check each layer:
- Welcome message. Does it say clearly that the visitor is talking to an AI assistant and what it can help with? (You can set this in the widget's appearance settings.)
- Answers. Are they supported by your sources?
- Detail collection. Are the lead fields and quote fields the ones your team needs (lead capture)?
- Privacy wording. Does the visitor know why details are being collected, with a link to your privacy policy?
- Booking path. Does the booking link open the right calendar (appointment booking)?
- Notifications. Does the email or SMS alert arrive, at the right address or number?
- Handoff. When a visitor asks for a person, does your team see it in Conversations, and what does the visitor see if nobody claims it in time (human handoff)?
- Transcript access. Can the person following up see the conversation?
Run the challenge on desktop and mobile, during business hours and after hours.
How to run it in Pivra#
- Add your sources and wait until each shows Trained.
- Use Chatbot → Test to run the 30 questions before publishing (improving responses).
- Fix failures at the source: correct or remove the outdated page or document, add plain-text facts, turn on strict mode if answers must stay within approved material, and use refinements for wording that must be exact.
- Enable lead capture and handoff (Starter plan and above) and send test leads and test handoff requests to confirm alerts arrive.
- Re-run the full set, then publish.
Keep it repeatable#
Keep the question set. Re-run it after a price change, a new service area, a model change or a large website update, and track regressions: answers that were right last time and are wrong now.
Over time, split the questions in two:
- a stable regression set you run every time, so results are comparable;
- a private adversarial set you change regularly, so the assistant is not simply tuned to known questions.
If customers or staff spot an edge case, add an anonymised version to the set.
Publish the method and the failures#
If you share results publicly, as a business or as a vendor, publish the method with the score: the categories, how answers were scored, the pass bar and the date. Publish the failures and what you changed, not only the final number. A score without a method is not evidence.
Pivra's setup already encourages testing answers before installing. The challenge makes that step concrete, repeatable and harder to skip.
Next step: copy the six categories into a spreadsheet, write your 30 questions and expected behaviours today, and run them before your chatbot goes live.
Topics
- AI chatbot launch test
- Chatbot testing
- AI chatbot quality
- Website chatbot
- Human handoff
- Lead capture testing
Author
The Pivra Team
Pivra
The Pivra team writes from product, onboarding, and customer support work with Australian tradies, local services, and small business operators.