The failure mode nobody demos
Vendor slides show a PDF uploaded and perfect answers flowing instantly. Production RAG over internal wikis, tickets, and policy libraries fails differently: answers sound confident but cite the wrong version of a procedure, or blend two departments' guidance into one unsafe instruction. The model is often fine; the retrieval layer lied.
Chunking by fixed token windows breaks tables, numbered steps, and exception clauses. A 512-token slice through an escalation matrix loses the row headers that make it meaningful. Structure-aware chunking—by heading, by ticket field, by slide title—costs more engineering but cuts nonsense retrieval sharply.
Embeddings are not magic synonyms. Acronyms, product codenames, and internal project names cluster poorly unless you enrich metadata. Add document type, effective date, owning team, and sensitivity label to every chunk before you debug the LLM prompt.
Access control is part of retrieval
If your vector database does not enforce the same ACLs as SharePoint or Confluence, you built a leak machine with conversational UI. Filter at query time using the user's groups, not at index time with a single super-user crawler. Pen testers will ask the chatbot for HR data; plan for that on day one.
Stale content is another ACL problem. Old policy chunks rank highly because they repeat generic phrases. Boost recency or maintain explicit version pointers in metadata. When legal updates a retention rule, expire embeddings from superseded PDFs instead of hoping users notice dates in footers.
Cross-document synthesis without citations trains users to trust tone over traceability. Require chunk IDs in answers and render clickable source links in the UI. Editors reviewing AI-assisted support macros need to see which paragraph justified each sentence.
Evaluation that catches regressions
Build a golden set of 50–100 real questions your teams already ask, with accepted answers and mandatory source IDs. Run it after every embedding model change, chunk size tweak, or reindex. Track precision@k and citation accuracy separately; high recall with wrong citations is worse than admitting ignorance.
Include adversarial prompts: requests for credentials, instructions to ignore policy, and questions outside the corpus. The system should refuse or escalate, not improvise. Log failures for human review instead of silently fine-tuning on bad interactions.
Human reviewers should score usefulness, not fluency. A polished paragraph that solves the wrong problem is a production incident waiting for a busy Monday.
Practical rollout sequence
Start with a narrow corpus: one product line, one support tier, or one compliance domain. Prove citation quality there before connecting the entire drive. Parallel runs against search-only baselines keep expectations honest.
Instrument latency end to end. Users tolerate slower answers if sources are visible; they do not tolerate three-second waits for vague summaries. Cache embeddings for hot documents but invalidate on change events from the source system.
RAG over company documents succeeds when retrieval engineers own the problem as much as prompt engineers. Fix chunk boundaries, metadata, ACL filters, and evaluation first; swap the LLM last.
Hybrid search beats pure vectors alone
Keyword filters rescue proper nouns that embeddings miss. Combine BM25 or equivalent lexical search with vector retrieval, then rerank with a cross-encoder when latency allows. Teams that skip lexical layers see more confident wrong name answers in HR and legal corpora.
Table-aware parsers matter for spreadsheets attached to tickets. Flattening a workbook to plain text destroys relationships between columns. Preserve sheet and cell references in chunk metadata so answers can point to row and column references instead of a mushy paragraph.
Redaction before indexing is irreversible if done carelessly. Automated PII scrubbers may remove names you still need for retrieval. Use role-based indexing: sensitive fields stay out of the vector store but can be fetched at answer time through authorized APIs.
Week-one incident: the wrong policy version
A support team wired RAG to a Confluence space that mixed draft and published pages. A customer received refund steps from a 2022 draft that legal had never approved. The chatbot cited a real page ID, so the agent trusted it. Fix: index only labels tagged `published`, store `effective_date` in metadata, and block answers when the top chunk predates the current policy banner date.
FAQ for legal and IT reviewers
Can we delete embeddings when a document is archived? Yes—treat de-indexing as part of records retention, not an optional cleanup.
Do we need human review on every answer? No, but high-risk intents (refunds, medical, safety) should require citation coverage above a threshold or escalate.
Is fine-tuning cheaper than fixing retrieval? Usually not for policy libraries; bad retrieval in a fine-tuned model fails more quietly.
Reindex playbook after a reorg
When teams rename products, run an alias table in metadata before re-embedding. Map old codenames to new SKUs for six months. Schedule a full reindex on weekends with shadow traffic comparing answer diffs. Alert when citation URLs 404.
When to pause the chatbot
Pause if precision@5 on the golden set drops twenty points, if ACL tests fail, or if more than three wrong-policy tickets arrive in one shift. A visible banner beats silent wrong answers.
Multilingual and legacy format pitfalls
Scanned PDFs in languages the embedding model under-represents retrieve poorly unless you specify language metadata and use multilingual embedders. Legacy formats—Lotus Notes exports, `.doc` binaries—need conversion pipelines with checksum validation; silent OCR failures produce empty chunks that still answer confidently.
Cost governance for embedding sprawl
Re-embedding entire drives monthly burns GPU budget. Incremental indexing on file-change webhooks costs engineering upfront but pays off above ten terabytes. Archive cold documents to cheaper object storage without vectors until queried.
Stakeholder demo discipline
Never demo RAG on production data in client meetings. Use sanitized subsets; one leaked salary table in a projector room ends careers. Prepare refusal examples showing the system saying "I don't know"—that builds more trust than a lucky guess.
## Executive search and board materials
Board packs and M&A rooms need higher ACL precision than engineering wikis. Segment indices by clearance level; never rely on prompt instructions alone to refuse classified queries. General counsel should red-team with questions attackers would ask after insider threat.
Decommissioning retired products
When products sunset, embeddings of old runbooks still answer as if supported. Run tombstone jobs that replace answers with migration pointers to successor docs. Support macros referencing retired SKUs cause chargebacks when chatbots quote obsolete refund rules.
## Procurement questions for RAG vendors
Ask: How do you enforce ACL at query time? Can we export chunk IDs with answers? What embedding model pin do you support? How do deletes propagate? SLA for reindex after bulk delete? Vague answers predict production pain.
Support macro integration
When agents paste AI suggestions into replies, require chunk links in the CRM widget—macros without sources should be disabled by policy, not honor system.
Enterprise RAG FAQ
Chunk size rule? No universal—test on your tables and lists.
ACL at query time? Mandatory for internal docs.
Delete embeddings on archive? Yes—as retention policy.
Hybrid search? Often yes for proper nouns.
Fine-tune first? Usually fix retrieval first.
Citation required? Yes for trust and audit.
Golden set size? 50–100 real questions minimum.
Pause bot when? ACL fail or precision drop.