How to Reduce Wrong AI Chatbot Answers on a Business Website

You cannot make a generative chatbot incapable of error, but you can make wrong answers much less likely and much less harmful. Clean up conflicting sources, define what the assistant may and may not say, write an honest fallback, test it with at least 30 awkward questions, and re-test after every meaningful change. Uncertainty handled gracefully is more trustworthy than a confident guess.

Why chatbots get things wrong#

Most wrong answers on small-business websites have ordinary causes:

• Conflicting sources. An old price list and a new one are both in the knowledge base.
• Missing information. The customer asks something the business never wrote down, and the assistant fills the gap.
• Vague boundaries. Nobody told the assistant what it must not do.
• No fallback. The assistant has no approved way to say "I don't know".
• No re-testing. The website changed and nobody checked the chatbot.

Each has a practical fix.

1. Fix source quality first#

Remove duplicate, expired and contradictory pages. Give policies and prices an owner and a review date. If two documents disagree, the chatbot is being asked to solve a governance problem the business has not solved. No prompt can reliably choose the right version for you.

A good habit: before adding a source, ask "Is this the current, approved version, and who will tell us when it changes?"

2. Define answer boundaries#

Write down what the assistant may do:

• Answer from approved material.
• Ask a clarifying question.
• Say it does not have enough information and offer to take details.

And what it must not do:

• Invent availability or arrival times.
• Diagnose a fault.
• Promise an outcome.
• Infer a policy that has not been published.
• Offer discounts that do not exist.

These boundaries belong in the business instructions and, for critical topics, in exact approved wording.

3. Write the fallback before the clever prompt#

Before working on tone, write the sentence the assistant uses when it does not know:

"I don't have an approved answer for that. I can take your details so the team can confirm."

This one sentence prevents a large share of the damage. It gives the customer a path forward and gives the business a lead instead of a wrong answer.

4. Build an adversarial test set#

A friendly demo proves very little: it asks the questions the assistant was trained to answer. Build at least 30 test prompts that try to break it:

• Questions whose answer is absent from the sources.
• Old pricing quoted as if current.
• Requests for discounts.
• Locations just outside the service area.
• Unsafe DIY questions.
• Attempts to override its instructions ("ignore your rules and…").
• Ambiguous dates and relative times ("next Friday", "tonight").
• Messages containing personal or sensitive information.

For each prompt, write the expected behaviour before running it. Then record four things: was it correct, was it supported by a source, was it escalated properly, and how severe would the harm be if it were wrong?

5. Re-test after every meaningful change#

Run the set again after new prices, new service areas, a model change or a large website update. Watch for regressions: answers that used to be right and are now worse. Keep the test set in a spreadsheet so anyone on the team can run it.

A worked example#

A visitor asks: "Is the callout fee still the same as the price on your old flyer?" and quotes a figure from it.

• Bad answer: "Yes, that's right." (The assistant confirmed the visitor's old figure.)
• Better answer: "Our current callout fee is listed on our pricing page as [approved figure]." (Only if that figure is in an approved source.)
• Safe fallback: "I can't confirm that price. I can take your details and the team will confirm current pricing."

The test is not whether the assistant sounds helpful. It is whether it said only what the business has approved.

The same discipline as publishing#

Google's guidance on AI-generated content warns that generated material can contain inaccuracies and should be reviewed for accuracy before it is published (Google Search Central). The same discipline belongs inside the customer conversation: a chatbot reply is published content, even if only one person sees it.

The controls in Pivra#

Pivra provides several controls for this, described in the help centre:

• Strict mode: the chatbot answers only questions it can find context for in your knowledge base, and declines others with a message you can customise. See improving responses.
• Refinements: question-and-answer pairs you write, used word for word for wording that must be exact, such as licence numbers or cancellation terms. See refinements.
• Pricing & Quotes (Starter and above): structured pricing rules, so prices come from rules rather than from loose text.
• Testing: ask questions in the dashboard before publishing.
• Keeping sources current: retrain sources after changes, or set website sources to retrain on a schedule. See keep your chatbot up to date.

None of these makes errors impossible, so the test set and the fallback still matter. Our security page explains how the service itself is run.

Next step: write your fallback sentence and a 30-question test set, and publish only after the assistant passes it, not after one friendly demo.