Answer from documents with RAG without treating retrieval as proof
Retrieve relevant permitted passages, attach source references and allow the answer to say when the documents do not support a claim.
In brief
Retrieval-augmented generation supplies relevant document passages to an AI before it answers. It can help with a changing document collection, but the retrieved text and generated conclusion both need checking.
The answer is more useful when a reader can open the cited passage and see which statement it supports.
Prepare a small useful collection
Start with a few documents and real questions. Preserve headings, page references and update dates. Remove duplicate obsolete copies or mark their version explicitly. A noisy collection can make a semantic match look relevant while supplying the wrong policy.
Inspect retrieval separately
Before evaluating the final answer, check whether the returned passages contain the necessary fact. A question about an exception needs the exception paragraph, not only a general policy. Improve retrieval or document structure when evidence is absent instead of asking the model to sound more confident.
Make uncertainty usable
Require a source reference for substantive claims and allow “not covered by these documents.” Test contradictory versions and nonexistent answers. Keep the underlying source reachable for authorized readers; a citation that only repeats a filename is weak if nobody can inspect the supporting passage.
Before you act
Retrieval can miss the right paragraph, choose an obsolete version or expose a document the user should not see. RAG is neither automatic truth nor an authorization system.
What to do next
- Prepare readable documents with version, source and access information.
- Retrieve passages only from material the user is allowed to read.
- Ask for evidence-based answers with citations and test questions whose answers are absent.
