Short answer: Accuracy is not a property of the chatbot. It is a property of your content plus the system's willingness to admit uncertainty. A retrieval-based widget answering from well-written, current pages is reliable on the questions those pages cover; the same widget on vague or contradictory content will produce vague or contradictory answers. Anyone quoting you a single accuracy percentage is describing a benchmark, not your business.
Why the question does not have a number
Ask "how accurate is a chatbot" and the honest answer is "for which questions, on whose content, judged by whom?"
Accuracy on questions your website answers clearly is a different measurement from accuracy on questions it never addresses. And "wrong" hides at least four different failures that need four different fixes:
The answer contradicts your content. Rare in a grounded system and usually a retrieval problem — the right passage was not found.
The answer is fabricated. The model produced something plausible that no source supports. This is hallucination in the strict sense, and grounding is the defence.
The answer is stale. Faithfully drawn from a source that is no longer true.
The answer is incomplete. Technically correct but missing the qualifier that changes the decision — the fee that only applies outside your standard area, the lead time that doubles in summer.
Only the second is a model failure. The other three are content and process failures, which is good news, because you control content and process.
The four failures, with examples
Abstractions are easy to nod along to and hard to act on, so here is each failure as it actually shows up in a transcript. Assume a physiotherapy clinic whose website says initial assessments are sixty minutes and cost a published fee.
Contradiction. A visitor asks how long the first appointment is and the widget says thirty minutes. The website says sixty. Something went wrong in retrieval, possibly a passage from a different service page winning the relevance contest. The fix is structural: give the fact its own clearly headed section.
Fabrication. A visitor asks whether the clinic does dry needling. The site never mentions it. A poorly grounded system answers "yes, our therapists offer dry needling as part of a treatment plan" because that sentence is statistically plausible for a physio clinic. A grounded one says it is not sure and offers to put them through. Grounding reduces this sharply. It does not eliminate it, which is precisely why the corrections loop and guardrails exist rather than being optional extras.
Staleness. A visitor asks about availability and the widget repeats a "currently accepting new patients from September" line written last year. Every part of the system worked correctly. The source was wrong.
Incompleteness. A visitor asks the price and gets the published fee, which is correct, but not the detail that the fee differs for home visits. The answer is true and the customer is still going to be annoyed on the day. This is the most common failure and the least noticed, because nothing in the transcript looks wrong.
Naming which of the four you are looking at is most of the work, because each has a different fix. There is more on the mechanics in why chatbots give wrong answers.
What actually determines accuracy
Your content quality is the dominant variable. Specific, unambiguous, well-structured pages produce good answers. A page that says "competitive rates, tailored to your needs" cannot produce a correct answer about price because it does not contain one.
Retrieval quality is second. If the relevant passage is not retrieved, the model cannot use it. Chunking that splits a fact across two passages, or a page so long that the key sentence gets buried, both hurt. Clean headings and focused pages help more than people expect.
Grounding discipline is third. A system instructed to answer only from retrieved context, and to decline when the context does not support an answer, has a far better error profile than one allowed to fill gaps from general knowledge.
The model matters least. Every current frontier model is competent at reading a passage and summarising it. That is the task. Model choice is not where your accuracy comes from.
Worth saying plainly: grounding lowers hallucination risk, it does not remove it. A model can still overreach at the edges, particularly when a passage nearly answers the question. That residual risk is why a serious setup pairs grounding with guardrails, decline-and-route behaviour, and a corrections loop for the ones that slip through. Anyone selling you zero risk is selling you something else.
Uncertainty is an accuracy feature
The most useful thing a widget can say is "I am not sure — let me put you through to someone."
A system that guesses gets some of those guesses right, which looks better on a naive score and is far worse in practice. One confidently wrong answer about a policy or a price costs more than fifty honest declines, because the customer acts on it.
Clerkzo is set up to admit uncertainty instead of guessing, and to offer a human handoff at that moment — a fallback response, in glossary terms. That combination — decline plus route — is what makes an honest failure a decent customer experience instead of a dead end.
Note the asymmetry from the customer's side. An honest decline costs them thirty seconds and a callback. A confident wrong answer about your cancellation policy costs them a wasted journey and an argument at reception. Do not weight those two failures equally.
How to measure it for your own business
Do this rather than trusting a vendor figure.
Build a test set. Write down thirty to fifty real questions from your phone, email and enquiry form. Include the awkward ones: edge cases, pricing exceptions, things you do not offer.
Run them in the playground, which mirrors the live widget. Grade each answer correct, incomplete, wrong or declined. Do not grade on tone.
Read the pattern, not the score. Wrong answers cluster. If four failures are all about availability, you have one stale source, not four problems.
Fix and re-run. Recrawl after content edits, replace outdated files, and use corrections where the content is fine but the answer was not — a flagged correction becomes an authoritative override for that question.
Then repeat with real conversations. Every conversation is saved with a thumbs up or down rating, so your live traffic keeps producing the test set for free. That loop is the real mechanism behind chatbot training in a retrieval system: observe, correct, verify.
Building the test set properly
The quality of your measurement depends entirely on the questions you choose, and most people choose badly by picking the ones they know the site answers well. Aim for this mix instead.
- Ten easy ones. Hours, location, main services, how to book. These should be perfect. If they are not, fix your content before doing anything else.
- Ten real ones. Verbatim from your phone log or inbox, typos and all. Nobody types "what are your service offerings".
- Five awkward ones. Pricing exceptions, out-of-area requests, the thing you only do sometimes.
- Five you cannot answer. Things you do not offer, or that depend on account specifics. The correct result is a decline plus an offer of a human, and scoring that as a failure is a mistake people make constantly.
- Five adversarial ones. Off-topic requests, or someone asking it to ignore its instructions. You are testing scope, not knowledge.
Grade with a four-point rubric and nothing fancier. Correct: right and complete. Incomplete: right but missing a qualifier that would change the customer's decision. Wrong: contradicts your content or invents something. Declined: refused and offered a route to a human. Track the four counts, not a single percentage, because the mix is what tells you where to work.
A cadence for keeping it honest
Accuracy is not a launch task, it is a habit. This is the smallest version that works.
Before launch. Run the full test set in the playground. Fix the content gaps it exposes. Re-run. Do not go live on a set you have not passed.
Week one, daily. Read every transcript. There will not be many, and this is the richest fortnight of feedback you will ever get. Correct wrong answers as you find them.
Monthly, fifteen minutes. Read the thumbs-down conversations and skim the rest. Look for clusters rather than individual complaints.
Quarterly, thirty minutes. Re-run the whole test set, adding the quarter's new questions. Audit uploaded files and corrections for anything that has expired.
On every content change, and before anything seasonal. Recrawl, then spot-check the answers that change affects. Ask the holiday-hours question a week before your customers do.
What good looks like in practice
Realistic expectations for a well-maintained widget on decent content:
Questions your content answers clearly — answered correctly and consistently.
Questions your content answers ambiguously — answered ambiguously. The widget cannot be clearer than its source.
Questions your content does not cover — declined, with an offer to reach a person.
Questions requiring judgement, current stock, or account-specific information — routed to a human, because they should be.
If your widget is producing confident answers in the third and fourth categories, that is a configuration problem, not a triumph.
The trap of chasing a perfect score
Past a point, pushing accuracy higher means narrowing scope until the widget declines anything interesting. That is not a better product; it is a worse one with better metrics.
The target is not perfection. It is that the widget is right on the common questions, honest about the rest, and always one click from a person. Get that and the accuracy conversation stops being anxious. Our pricing is flat rather than per-conversation, which matters here: you are never tempted to suppress conversations to control cost.
There is a fairer benchmark than perfection, too. Compare the widget with what happens today, not with an ideal. At nine on a Sunday evening, the honest accuracy of your business is zero, because nobody answers. The widget is not competing with a flawless human. It is competing with silence.
Frequently asked questions
Can an AI chatbot be one hundred percent accurate?
No system dealing with open-ended natural language is. Aim for reliable on common questions and honest on the rest.
Why did it get something wrong that is clearly on my website?
Usually retrieval. The passage may be buried in a long page, split awkwardly, or contradicted elsewhere on your site. Restructure the page or add a correction.
Is a bigger model more accurate for my business?
Marginally at most. Reading a retrieved passage is not a hard task for current models. Your content is the lever.
How do I stop it inventing things?
Ground answers in your own retrieved content, scope it to your business, and configure it to decline rather than guess.
Should I count a decline as a failure?
No, count it separately. A decline on something you genuinely cannot answer is the system behaving correctly. A rising decline rate on things you do cover is the real signal, and it usually means a content gap rather than a model problem. What content should you train your chatbot on covers how to close those.