Preparing a chatbot knowledge base: from documents to testable answers

A chatbot does not become reliable because many documents are uploaded. Conflicting policies, outdated prices, scanned PDFs and unclear ownership can produce answers that sound convincing but are wrong. The knowledge base is an editorial system, not a file store.

Define a controlled scope

Specify which questions the bot may answer and which actions it may perform. Separate general information, account-specific data, recommendations and binding actions. The higher the impact of an error, the tighter the source, approval and handoff controls need to be.

Inventory sources

Record:

• document or URL;
• topic and audience;
• subject-matter owner;
• approval status;
• validity period;
• language and region;
• confidentiality level;
• last review date;
• superseded version.

A public website, internal policy and CRM may disagree. For each field, identify the authoritative system.

Clean the material

Remove duplicates, obsolete versions and contradictions. Convert scanned or complex files into reliably readable text. Preserve the meaning of headings, tables and lists.

Do not write only for one example question. Include terminology, synonyms, constraints, exceptions and required clarification. One FAQ pair may cover a simple issue; complex rules need structured articles.

Add metadata and boundaries

Tag product, country, language, customer type, validity and permission. This helps a system restrict retrieval to appropriate sources. Define subjects that require human review.

The bot should not fill uncertainty with invented detail. A controlled response can name missing information, ask a question or transfer to an authorised person.

Separate retrieval quality from writing quality

A wrong answer may result from selecting the wrong source or from summarising a correct source incorrectly. Record which documents supported each test case. This allows the team to decide whether to improve the content, metadata, retrieval process or response rules.

Create a fixed test set

Include:

• common questions;
• varied wording and spelling errors;
• combined requests;
• out-of-scope questions;
• conflicting or incomplete details;
• outdated terminology;
• sensitive or binding decisions;
• human-handoff cases.

Evaluate factual accuracy, source support, completeness, permitted voice and correct escalation. Fluency alone is not a quality measure.

Plan maintenance

Sources change. Define who updates them, when tests rerun and how old versions are removed. Critical policies need an expiry date or recurring review. Record changes so failures can be traced to a version.

Run a controlled pilot

Start with a narrow task and known users. Monitor unresolved questions, wrong sources, repeated clarification and transfers. Expand only after the current scope is stable.

Acceptance gate

The knowledge base is ready when every active source has an owner and status, conflicting versions are removed, sensitive boundaries are defined, a representative test set passes, and maintenance and escalation have named owners.

Reliable chatbot answers do not begin with the model. They begin with approved, maintainable information.