RAG Architecture: Enterprise Design Guide

Retrieval-augmented generation looks simple in a demo: split documents, create embeddings, retrieve a few passages, and place them in a prompt. Enterprise RAG architecture is a different problem. The system must keep thousands or millions of changing knowledge objects searchable, preserve source permissions, explain every answer, and recover when content, identities, models, or indexes change.

That changes the architectural question. The goal is not merely to connect a vector database to a language model. The goal is to build a governed knowledge system that can ingest authoritative content, retrieve the right evidence for each user, assemble bounded context, and continuously prove that the result remains correct.

Four-plane enterprise RAG architecture connecting source systems to ingestion, query, policy, evaluation, and cited answers

The short answer

A production RAG architecture should be designed as four connected planes:

Ingestion plane: captures source changes, parses content, creates durable chunks, enriches metadata, and maintains indexes.
Query plane: understands the request, retrieves across lexical, vector, structured, and graph signals, reranks evidence, and assembles context.
Policy plane: carries identity, access control, provenance, retention, and audit rules through ingestion and retrieval.
Evaluation and operations plane: measures retrieval and answer quality, traces failures, controls releases, and repairs stale or unsafe knowledge.

The model is only one component. In practice, the difficult work is maintaining the contract between the source of truth, the retrieval layer, the user’s permissions, and the answer that appears on screen.

Why enterprise RAG needs a system architecture

The familiar RAG pipeline—retrieve, augment, generate—describes a request path, not an operating model. It leaves out most production failure modes:

A policy changed in the source system, but the old version remains in the index.
A user lost access to a document, but its chunks are still retrievable.
A PDF table was parsed as disconnected text and now produces a misleading answer.
Two sources disagree, and the system cannot identify which one is authoritative.
A new embedding model improves a benchmark but silently changes retrieval for real users.
A source document is deleted, yet its derivative chunks, vectors, summaries, and caches survive.

Microsoft’s current RAG guidance calls out query understanding, multi-source access, token constraints, latency, security, and governance as first-class challenges. It also recommends hybrid retrieval, structured responses, citations, incremental indexing, and document-level security controls rather than treating vector similarity as the whole solution. Microsoft: RAG in Azure AI Search

The useful design unit is therefore not the prompt. It is the knowledge object lifecycle: where an object came from, how it changed, which representations were derived from it, who may retrieve it, how it was selected, and what answer used it.

A four-plane reference architecture

The four planes separate responsibilities without isolating them. Every query crosses all four, and every content change must be reflected across all four.

1. Ingestion plane

The ingestion plane turns heterogeneous source material into retrievable, governed knowledge objects. It usually includes:

source connectors and webhooks
change capture and crawl scheduling
file-type detection and malware scanning
OCR, layout-aware parsing, and table extraction
normalization and duplicate detection
chunking and parent-child relationships
metadata enrichment and entity extraction
embedding generation
lexical, vector, structured, and graph indexing
retry queues, dead-letter handling, and freshness monitoring

Chunking deserves architectural attention because it affects retrieval, citations, permissions, and deletion. Amazon Bedrock documents fixed-size, hierarchical, semantic, and no-chunking strategies, while preserving a mapping from each chunk to its original document. That mapping is not optional bookkeeping; it is what makes provenance and lifecycle repair possible. Amazon Bedrock: content chunking

The ingestion plane should assign stable identifiers at more than one level:

source_id for the connected system
document_id for the logical object
version_id for a source revision
chunk_id for a retrievable unit
derivation_id for embeddings, summaries, captions, or extracted entities

Without stable identities, re-indexing becomes duplication, deletions become best-effort, and citations cannot reliably return to the exact evidence used.

2. Query plane

The query plane turns a user request into a compact evidence set. A robust path may include:

authenticate the requester and establish tenant, role, and session context
classify intent and decide whether retrieval is needed
rewrite, expand, or decompose the query when useful
select eligible sources or indexes
apply permission filters before or during candidate retrieval
retrieve candidates using multiple signals
fuse and rerank the candidates
remove duplicates and resolve parent-child context
assemble a token-bounded evidence package
generate an answer with citations
run output policies and record a trace

Hybrid search is usually the default starting point because enterprise questions contain both concepts and exact identifiers. Vector retrieval can match paraphrases, while lexical retrieval can preserve product codes, policy names, error messages, people, and dates. Azure AI Search, for example, runs full-text and vector queries in parallel and merges their rankings with reciprocal rank fusion. Microsoft: hybrid search

Structured and graph retrieval add another dimension. A question such as “Who owns the renewal, what was decided, and which policy governs the exception?” may require a customer record, a meeting decision, an owner relationship, and a policy passage. Similarity search alone does not reliably join those facts. A production query plane should be able to route to documents, tables, applications, and relationship indexes, then preserve the role of each item in the final evidence set.

3. Policy plane

The policy plane is the difference between useful enterprise retrieval and an accidental data-exfiltration system. It should govern:

tenant isolation
user and group identity
document- and row-level permissions
source authority and trust tier
geographic and regulatory boundaries
retention and legal hold
prompt-injection handling
sensitive-data controls
audit logging
action authorization for agents

Permissions must travel with the object, not be added after retrieval. A secure implementation either evaluates the source system at query time or copies normalized access-control metadata into the index and filters candidates against the current requester. The choice depends on latency, source capabilities, and the acceptable freshness window, but the result must be fail-closed.

Security trimming also has to survive every representation. If one document produces twenty chunks, an abstract, three extracted entities, and a cached answer, revoking the document must revoke all of them. The policy plane therefore needs an explicit derivation graph, not just an ACL field on the original file.

4. Evaluation and operations plane

RAG quality changes whenever content, parsers, chunking rules, embeddings, ranking weights, prompts, or models change. Evaluation is therefore a release system, not a one-time benchmark.

The operations plane should capture:

representative question sets by role and task
expected sources or evidence attributes
retrieval recall and precision
rank quality and context utilization
citation correctness
groundedness and completeness
abstention behavior
permission and adversarial tests
latency and cost budgets
end-to-end query traces
user feedback tied to reproducible cases

Google’s reference architecture for RAG separates serving from evaluation and describes evaluation outputs that include model responses, metric scores, and optimized system instructions. The broader lesson is to version the entire candidate—not just the model—so a quality change can be traced to the exact parser, index, retrieval configuration, prompt, and model that produced it. Google Cloud: RAG infrastructure

The ingestion architecture: from source change to governed index

Enterprise ingestion should be incremental and event-aware. Full recrawls are useful for reconciliation, but they are too slow and expensive to be the primary freshness mechanism.

A practical lifecycle looks like this:

Detect: receive a webhook, change token, event-stream record, or scheduled crawl result.
Fetch: retrieve the source object and its current permission state using a scoped connector identity.
Validate: confirm type, size, tenant, integrity, and allowed processing policy.
Parse: preserve headings, lists, tables, captions, pages, and layout relationships.
Normalize: produce a canonical document model while retaining links to original offsets.
Segment: create retrieval units aligned with semantic and structural boundaries.
Enrich: add title, section path, dates, owners, entities, source authority, and other useful fields.
Index: write lexical terms, vectors, structured fields, relationships, and ACL metadata.
Verify: read back counts and sample objects, then mark the version searchable.
Retire: remove or tombstone superseded derivatives after the new version is proven ready.

This sequence suggests a two-phase publish model. Build the new derivative set under a version identifier, verify it, and only then move the searchable pointer. Users should not see half of a document indexed under new rules while the other half remains under old rules.

Preserve structure before choosing chunk size

Teams often begin by debating 300 versus 800 tokens. The more important question is whether the parser preserved the object’s meaning. A policy section, troubleshooting procedure, table row, source-code function, meeting decision, and CRM record have different retrieval boundaries.

Useful chunk schemas often include:

the text or structured payload
document and version identifiers
parent and child chunk links
heading and breadcrumb path
character or page offsets
content type and language
created, updated, effective, and expiry dates
source URL and source authority
owner or steward
access-control principals
parser, chunker, and embedding versions

Metadata enables filtering and ranking without forcing every distinction into the embedding. Microsoft’s chunk-enrichment guidance similarly recommends cleaning and augmenting chunks with fields such as titles, summaries, keywords, and tags to support retrieval beyond semantic similarity. Microsoft: RAG chunk enrichment

Retrieval architecture: recall first, then precision

Retrieval has two jobs that should be measured separately:

Candidate generation should avoid missing the relevant evidence.
Reranking and context assembly should remove noise and preserve the best evidence within the token budget.

A common enterprise pattern is:

query understanding → source routing → parallel retrieval → permission filtering → rank fusion → reranking → diversity and authority rules → context assembly

Candidate retrieval can combine:

BM25 or other lexical matching
dense-vector similarity
sparse semantic vectors
metadata and time filters
entity and relationship expansion
structured queries against tables or applications
curated authoritative results

Reranking can use a cross-encoder, a language model, business rules, or a cascade. Whatever the method, keep the pre-rerank candidates and scores in the trace. Otherwise, a bad answer cannot be diagnosed as a recall failure, ranking failure, context failure, or generation failure.

Amazon Bedrock exposes retrieval separately from retrieve-and-generate and supports reranking, source chunks, citations, query generation for structured stores, and iterative agentic retrieval traces. That separation is a useful architectural principle even when using another stack: retrieval should be observable and testable independently of answer generation. Amazon Bedrock: retrieving from knowledge bases

Context assembly is an explicit subsystem

Do not simply concatenate the top ten chunks. Context assembly should decide:

whether adjacent chunks are needed
whether a parent summary adds necessary scope
whether multiple results repeat the same claim
whether the evidence represents conflicting versions
whether a lower-ranked authoritative source should outrank an informal one
whether the question requires a table, relationship, or live application record
how much evidence can be included without crowding out instructions and the user request

The output should be a structured evidence package with stable source identifiers, quoted spans or fields, authority metadata, timestamps, and citation labels. The generator should not need to guess where one source ends and another begins.

Permission-aware retrieval by construction

Adding a security filter at the end of the query is not enough. The architecture must answer four questions:

Which identity is the retrieval performed for?
When was the permission state last synchronized?
Which derived objects inherit or transform the original permission?
What happens when the permission service or source connector is unavailable?

The safe default is to return less, not more. If an ACL cannot be evaluated, the object should be ineligible. If a shared summary contains material from documents with different permissions, the summary should carry the intersection or be generated per user—not inherit the broadest source permission.

Multi-tenant deployments also need a physical and logical isolation decision. Options include separate indexes per tenant, shared indexes with mandatory tenant filters, or hybrid partitions by risk tier. Shared infrastructure can reduce cost, but every query path, cache, background job, and evaluation harness must preserve tenant context. A single missing filter becomes a cross-tenant incident.

Content lifecycle diagram tracing one source through parsed text, chunks, vectors, entities, summaries, and verified deletion

Freshness, deletion, and the derivative-data problem

RAG systems create more copies than teams expect. One source object can produce parsed text, thumbnails, OCR output, chunks, vectors, summaries, entities, graph edges, query caches, generated answers, and evaluation fixtures.

That creates three operational contracts:

Freshness contract

Define how quickly a source update must become searchable. A handbook may tolerate hours; an incident runbook or customer entitlement may require minutes or live lookup. Track freshness by source and object class, not as one global number.

Deletion contract

Define the maximum time to remove every searchable or reusable derivative after deletion or access revocation. Use tombstones and an asynchronous cleanup ledger so removal is observable and retryable. Verify absence after cleanup instead of assuming a successful queue message means the data is gone.

Repair contract

Define how the system detects and repairs drift: missed webhooks, connector failures, partial index writes, parser regressions, embedding-version mixtures, and orphaned chunks. Periodic source-to-index reconciliation is essential even when event delivery is reliable.

RAG evaluation trace isolating a reranking failure and routing it to a controlled repair candidate

Evaluation architecture: test evidence, not just prose

Answer-level scores can hide retrieval defects. A model may produce plausible prose from weak or wrong evidence. Evaluate the chain in layers:

Ingestion tests

Was the full source parsed?
Were headings, tables, and relationships preserved?
Are chunk boundaries useful for the intended tasks?
Do ACLs and lifecycle identifiers match the source?

Retrieval tests

Did at least one relevant item enter the candidate set?
Did the best evidence reach the top ranks?
Were exact identifiers and semantic paraphrases both handled?
Did time, authority, tenant, and permission filters behave correctly?

Context tests

Is the selected evidence sufficient and nonredundant?
Are conflicting versions identified?
Do citations resolve to the exact source span or structured record?

Answer tests

Is every material claim supported?
Did the system distinguish facts from inferences?
Did it abstain when evidence was missing or inaccessible?
Did it avoid following instructions embedded in retrieved content?

Operational tests

Can the query be reproduced from its trace?
Does the candidate meet latency and cost budgets?
Can rollback restore the previous parser, index, retrieval, prompt, and model combination?

A useful release gate includes both offline evaluation and a shadow or canary phase. Offline sets provide reproducibility; production traces reveal language, sources, roles, and edge cases that the benchmark missed.

Common RAG architecture mistakes

Treating the vector database as the architecture

A vector index is one retrieval component. It does not solve change capture, authority, ACLs, structured data, citations, evaluation, deletion, or repair.

Copying source content without a lifecycle model

If the index cannot identify the current document version and all its derivatives, it will eventually serve stale or revoked knowledge.

Using one chunking rule for every content type

Uniform token windows destroy procedures, tables, records, and code boundaries. Start with the source object model and task, then choose segmentation.

Evaluating only generated answers

Without retrieval and context metrics, teams tune prompts to compensate for missing evidence. The result is fragile and hard to debug.

Applying permissions after candidate retrieval

This can leak sensitive data through traces, caches, rerankers, or model context even if the final UI hides the source. Enforce eligibility before content reaches downstream components.

Making citations decorative

A citation should identify the exact source object, version, and supporting span. Linking to a homepage or a mutable document without version context does not make an answer auditable.

Build or buy: what should remain yours

Managed services can accelerate connectors, parsing, embeddings, indexing, hybrid search, reranking, and model access. The durable architecture still needs product-specific decisions that should remain portable:

canonical knowledge-object schema
identity and permission semantics
source authority model
retrieval and answer evaluation sets
trace format
freshness, deletion, and repair contracts
user feedback taxonomy
release and rollback policy

This boundary avoids two extremes: rebuilding commodity infrastructure and outsourcing the organization’s knowledge model to one vendor-specific index.

A practical rollout sequence

Phase 1: one bounded knowledge domain

Choose a task with clear authoritative sources, known users, and measurable questions. Establish stable identifiers, citations, and permission checks before expanding connector count.

Phase 2: instrument the retrieval chain

Record candidates, filters, scores, reranking, context selection, generation settings, and citations. Build a small but difficult evaluation set from real work.

Phase 3: add hybrid and structured retrieval

Introduce lexical, vector, metadata, and application queries based on observed failures. Do not add complexity without a failure class and evaluation target.

Phase 4: formalize lifecycle operations

Set freshness service levels, deletion verification, source reconciliation, re-indexing controls, and rollback procedures.

Phase 5: expand sources and agent behavior

Only after the knowledge layer is observable and governed should agents dynamically choose sources, iterate retrieval, or act on retrieved information.

The architecture test

A credible enterprise RAG architecture should answer these questions without hand-waving:

Can every answer be traced to exact, authorized evidence?
Can every indexed object be traced to a source version?
Can a deletion or permission revocation remove all derivatives?
Can retrieval be evaluated independently of generation?
Can the system explain whether a failure came from ingestion, retrieval, context, policy, or generation?
Can a new parser, embedding, index, reranker, prompt, or model be released and rolled back as one versioned candidate?

If the answer is no, the system may still be a useful prototype. It is not yet a dependable enterprise knowledge system.

Final takeaway

The strongest RAG architecture is not the one with the most models or retrieval tricks. It is the one that keeps knowledge, permissions, provenance, evaluation, and operations connected as the system changes.

Design the four planes together. Preserve stable object identities. Retrieve with multiple signals. Enforce permissions before downstream use. Treat citations, deletion, evaluation, and repair as architecture—not polish. That is how RAG becomes a trustworthy knowledge layer for people and agents rather than a demo that slowly drifts away from its sources.

Architecture decision matrix

Layer	Required control	Launch evidence
Ingestion	Source identity, change capture, and deletion propagation	A deleted source disappears from every derivative index
Retrieval	Hybrid recall with permission filters before sensitive processing	Unauthorized candidates never reach reranking or generation
Context	Traceable evidence assembly with citation boundaries	Every answer span maps to an authorized source
Operations	Evaluation, freshness SLOs, replay, and repair	Failures can be reproduced and corrected without a full rebuild

Frequently asked questions

What is the minimum viable enterprise RAG architecture?

Start with one bounded source domain, identity-aware ingestion, permission-filtered retrieval, explicit context assembly, citations, and an evaluation set. Add graph retrieval or agents only after this path is observable.

Should permissions be applied before or after vector search?

Before sensitive content reaches reranking, context assembly, or generation. Post-filtering can expose unauthorized material to downstream components and can reduce result quality unpredictably.

How should an enterprise RAG system handle deleted content?

Treat deletion as a propagated event. Remove or tombstone the source object, chunks, embeddings, cache entries, graph edges, and evaluation fixtures, then verify absence.

Is a vector database the core of RAG architecture?

It is one retrieval component. The architecture also needs connectors, identity, parsing, lifecycle controls, evaluation, observability, and answer policy.

What should teams measure first?

Measure permission correctness, retrieval recall, evidence usefulness, freshness, no-answer behavior, latency, and cost per successful task—not fluency alone.

Where Dokki fits

Dokki is useful when the work needs a shared, reviewable knowledge layer rather than another isolated chat. Teams can keep source documents, decisions, structured trackers, and agent work in one workspace, then publish only the approved output. A practical next step is to model one bounded RAG workflow in Dokki and use its documents and tables as the evaluation and operating record.

Sources

_Last verified: July 21, 2026._

RAG Architecture: Enterprise Design Guide | Dokki