Retrieval-augmented generation (RAG) gives a language model selected source material to help answer a question. It does not guarantee a correct answer. A useful implementation needs relevant documents, permission checks, evaluation of the final answer, and a defined response when the evidence is missing or contradictory.
If you are commissioning a knowledge assistant, start with a decision: what questions should it help people resolve, and what would make an answer unacceptable? This guide sets out our recommended scoping questions and acceptance checks. It is not a customer case study or a promise of a particular accuracy, price, or delivery date.
When RAG fits—and when a simpler workflow is enough
Consider RAG when users need to find and combine information across a changing collection of documents. For example, an internal support team might need answers tied to the current operating procedure rather than a model’s general knowledge. That is an illustrative use case, not a DataBackfill customer result.
If users already know which document they need, better search or navigation may be enough. If the answer is a current account balance or order status, use the authoritative application API with appropriate authorization. RAG over exported documents should not silently stand in for live transactional data. Separate finding information from taking actions such as updating a record or sending a message.
Define the pilot before choosing a vector database
- Name one audience, one bounded question set, and the person who can judge answer quality. Record the current manual process so the pilot has a meaningful comparison.
- Inventory approved sources, owners, formats, update frequency, and who is allowed to read each source. Start with non-sensitive examples during initial scoping.
- Agree what a successful answer must include: source passages, relevant qualifications, and a clear indication of uncertainty or missing information.
- Define out-of-scope questions, escalation ownership, and the conditions that would stop or delay rollout.
Ask a prospective implementation partner to show how these decisions become test cases and deliverables. A fluent demonstration is useful for feedback, but it does not establish performance on your documents, users, or permission boundaries.
Compare retrieval approaches on your questions
Start with a baseline and change one retrieval choice at a time. Examine whether the needed passages were retrieved before blaming the generator. Preserve document identifiers, version information, headings, and relevant table context so reviewers can trace an answer back to its source.
Anthropic’s Contextual Retrieval research describes combining semantic retrieval with lexical matching, adding context to chunks, and reranking candidates. Its experiments support testing these techniques; they do not establish which configuration will work best for your corpus. Reranking also introduces a runtime cost and latency tradeoff.
Source: Anthropic — Introducing Contextual Retrieval
Our recommendation is to compare alternatives on the same representative questions and report retrieval quality alongside end-to-end response time and cost. Include exact identifiers, paraphrases, tables, and questions whose answers span documents. Do not assume that one chunk size, a larger model, or more retrieved passages is always better.
Treat document permissions and freshness as acceptance criteria
A login screen does not by itself establish which documents a signed-in user may retrieve. Microsoft’s Azure AI Search documentation describes query-time document filtering and permission-aware approaches. It also documents preview features and synchronization limits; the available controls depend on the selected integration.
Source: Microsoft — Document-level access control in Azure AI Search
For the project you are buying, require a demonstrated permission boundary before restricted text reaches the model. Test with users who have different roles, a revoked permission, a deleted document, and a stale indexed copy. Ask how source changes reach the index and what happens while synchronization is delayed. Include retrieved passages, caches, logs, and displayed citations in the access review.
Agree who owns document corrections and how to withdraw a bad source. A response can faithfully summarize an outdated policy and still mislead the user. Keep source-version information available for investigation, and define which freshness failures require an answer to be withheld.
Evaluate the answer, not just the retrieved passage
Microsoft’s RAG evaluation documentation separates retrieval quality from answer groundedness, relevance, and completeness. These measures ask different questions: useful context does not establish that the final response used it correctly, and a source-supported response can still omit information the user needs.
Source: Microsoft — Retrieval-Augmented Generation evaluators
Our recommended evaluation set includes answerable questions, missing evidence, conflicting versions, unauthorized requests, and misleading instructions embedded in source documents. Ask domain reviewers to record expected supporting passages and unacceptable outcomes. Treat retrieved material as data rather than permission to change system rules or perform actions.
Check that each citation actually supports the nearby claim. Define when the assistant should ask a clarifying question, decline, or refer the user to a named team. Review both incorrect answers and unnecessary refusals. Use automated scores to help find regressions, with human review for consequential decisions and disputed cases; a single aggregate score is not a release decision.
What drives RAG implementation cost?
Separate one-time engineering from ongoing operation. An estimate should state its assumptions rather than hiding them behind a fixed number that ignores your data and access model.
- Implementation: connector work, document parsing and cleanup, identity integration, evaluation design, user interface, and deployment requirements.
- Operation: indexing and re-indexing, search storage and requests, model inputs and outputs, optional reranking, and expected usage peaks.
- Ownership: monitoring, investigation of wrong answers, source maintenance, security review, and regression testing when models or retrieval settings change.
To compare proposals, ask each partner to price the same pilot boundary and identify exclusions. Request a plan for measuring cost per completed useful task, not only cost per model call. That recommendation requires your team to define what counts as useful; it is not a benchmark or a savings claim.
What to ask for before production handover
- A versioned evaluation set and results, including failures and the agreed rollout decision.
- Documented data flows, permissions, source-update handling, and the boundaries of any external processing.
- An operational owner, monitoring and escalation instructions, and a tested way to disable the assistant or fall back to existing search.
- Ownership of code and configuration, access to deployment documentation, and a change-review process for prompts, models, and retrieval.
Start with a limited audience and compare the pilot with the existing workflow. Measure whether people complete the intended task, where they still need help, and whether the operational burden is acceptable. Expand only when the evidence and responsible owners support it.
Discuss a bounded RAG implementation
DataBackfill offers RAG and knowledge-system implementation alongside AI agent and workflow engineering. Bring the workflow, source-system names, access constraints, and examples of the questions your team needs answered. Do not include credentials, private documents, or personal data in the initial inquiry. Those details are enough to begin discussing fit and what a scoped evaluation should cover.