Ask any employee where the current version of the travel policy lives and watch what happens. The confident ones name a location and are wrong about a third of the time. The rest message a colleague, which is why a meaningful fraction of internal communication volume consists of people asking other people questions that a document already answers.
The organizational cost is not the two minutes it takes to ask. It is the interruption on the other end, the inconsistency when two people give different answers, and the decisions made on stale information because someone found a 2023 version of a document that was superseded twice. Traditional enterprise search was supposed to fix this and largely did not, because keyword search returns twelve documents when the user wanted one sentence.
AI knowledge bases change the shape of the answer. Instead of a ranked list of files, the system returns a direct response synthesized from the source content, with citations back to the documents it drew from. That difference sounds cosmetic and is not — it moves the burden of reading, comparing, and reconciling from the employee to the system. But the technology is only the visible tenth of the project. What determines whether the thing gets used six months after launch is a set of decisions about content, retrieval, and permissions that most teams make implicitly and should make deliberately.
Ingestion Is Seventy Percent of the Work
The demo ingests a folder of clean PDFs. The production system ingests a wiki with 4,000 pages of which 900 are drafts, a shared drive with eleven copies of the same handbook, a ticketing system where the real answers live in resolution notes, a set of Slack channels, and a folder of PowerPoints from an all-hands three years ago.
Three decisions matter here more than any model choice.
Decide what is authoritative before ingesting anything. If both HR_Policy_2024_FINAL.docx and HR_Policy_v3_revised.docx are in scope, the system will cheerfully synthesize an answer from whichever it retrieves, and the user has no way to know it got the wrong one. Source curation is not a nice-to-have. A smaller corpus of confirmed-current content produces dramatically better answers than a comprehensive corpus containing contradictions.
Preserve structure during extraction. A policy document's meaning depends heavily on its hierarchy — which subsection a rule sits under, what the parent heading scopes it to. Naive text extraction flattens all of that and produces chunks that read as universal rules when they were conditional. Tables suffer worst: a benefits eligibility matrix converted to a stream of text is actively misleading.
Handle the formats that carry real information. In most organizations, a surprising amount of operational knowledge lives in spreadsheets and slide decks rather than prose documents. Both extract poorly by default and both need specific handling. Teams that have already built disciplined document management for compliance have a large head start here, because the hard work of establishing which document is canonical is already done.
Chunking and Retrieval, Without the Mysticism
Retrieval works by splitting source content into passages, converting each into a vector representation of its meaning, and finding the passages closest in meaning to the user's question. The chunking strategy — how you split — has more effect on answer quality than most teams expect.
Fixed-size splitting at, say, 500 tokens is the default and the worst option for structured documents, because it cuts mid-table and mid-procedure. Splitting on document structure, so that a chunk corresponds to a section or a complete procedure, produces retrievals that make sense in isolation. Adding the document title and heading path as a prefix to every chunk costs almost nothing and materially improves both retrieval and the model's ability to answer with appropriate scope.
Two techniques are worth the extra engineering in most enterprise deployments.
Hybrid retrieval. Pure semantic search is weak on exact identifiers — error codes, part numbers, policy reference numbers, employee IDs. Running keyword search alongside vector search and merging the results recovers the cases where the user knows the exact term and semantic similarity does not help.
Reranking. Retrieve a wider candidate set, then use a second model to score each candidate specifically against the question. This is more expensive per query and consistently improves the precision of what reaches the answer-generation step.
Metadata filtering rounds it out. Restricting retrieval by department, document type, jurisdiction, or effective date lets a single knowledge base serve a question about German employment policy without pulling in the US handbook.
Permissions Are a Retrieval-Time Problem
This is where projects fail in ways that damage trust permanently, and the failure is architectural rather than accidental.
If access control is applied after retrieval — filtering the final answer — the model has already read content the user is not entitled to see, and traces of it leak through summarization and phrasing. The correct design filters at retrieval time: the user's identity and group memberships constrain which chunks are even eligible to be returned.
That requires permission metadata to travel with the content from ingestion onward, and to stay synchronized when source-system permissions change. A document reclassified as confidential in the source needs to become unretrievable in the index within minutes, not at the next full reindex.
Test this adversarially before launch. Have someone attempt to extract restricted information — compensation bands, an unannounced reorganization, an active legal matter — through indirect questions rather than direct ones. Models are helpful by disposition and will often assemble a restricted answer from unrestricted fragments. One incident where an employee surfaces salary data through a chat interface will end the project regardless of how well it performs otherwise.
Measuring Answer Quality Instead of Hoping
Most teams launch without an evaluation set and then argue about quality based on anecdotes. Build the evaluation set first. It takes about a day.
Collect 100 to 200 real questions — from the help desk queue, from internal chat channels where people ask colleagues, from onboarding FAQ lists. For each, have a subject matter expert write the correct answer and name the source document. That gold set becomes the regression test for every subsequent change to chunking, retrieval, prompting, or content.
Score three things separately, because they fail for different reasons:
Retrieval quality — did the correct source document appear in the retrieved set at all? If not, no amount of prompt tuning helps; the problem is ingestion or chunking.
Answer correctness — is the generated response factually right and appropriately scoped? A response that is technically accurate but omits a critical exception is wrong in practice.
Citation fidelity — do the cited sources actually support the claims made? This is the check that catches the most dangerous failure mode, where a plausible answer is stitched from unrelated passages and given authoritative-looking citations.
Track deflection rate too, but treat it carefully. Deflection tells you people stopped asking a human; it does not tell you they got the right answer. Pair it with a thumbs-down mechanism and, more importantly, with periodic sampling of real production questions scored by an expert.
Content Decay Is the Slow Killer
Every knowledge base is accurate on launch day and degrades from there. Policies change, systems are replaced, processes are revised, and the documents describing them are updated at wildly uneven rates.
The failure is quiet. The system confidently returns a procedure that was correct eighteen months ago, with a citation, and the employee follows it. Nobody reports a bug because nothing looked broken.
Three mechanisms hold this back. First, ownership: every source document has a named owner and a review interval, and unreviewed content past its interval gets flagged in answers or demoted in retrieval. Second, freshness signals surfaced to the user — showing when a source was last updated lets a human apply judgment the system cannot. Third, gap analysis from query logs. Questions that consistently produce poor retrieval are telling you what documentation does not exist. That log is the most useful content roadmap most organizations will ever get, and it costs nothing to read.
Organizations running structured HR audit and compliance cycles can attach knowledge base content review to the same calendar rather than establishing a parallel process nobody owns.
A Rollout That Survives Contact With Users
Launch narrow. One well-curated domain — IT support, HR policy, or a single product line — with genuinely current content beats an organization-wide rollout over an uncurated corpus. Users form a judgment about reliability in their first five queries, and that judgment is very hard to revise.
Put it where the questions already happen. A knowledge base that requires visiting a separate portal competes with messaging a colleague and loses. Embedded in the chat tool where people already ask each other, it wins by default.
Set expectations explicitly in the interface. Tell users what is in scope, what is not, and how current the content is. Systems that answer everything with equal confidence teach users to distrust all of it; systems that say "this is not covered in the available documentation" earn trust for the answers they do give.
The gap between a knowledge base people use daily and one that quietly dies is almost never the model. It is content curation, retrieval design, and whether anyone owns freshness. If you are scoping what that would take against your own documentation, an AI knowledge base built against one real domain and measured with a proper evaluation set will answer the feasibility question faster than a procurement process will — and the Workisy team is happy to talk through the scoping before anything gets built.
