Enterprise Search Architecture: A Production Guide | Dokki

Enterprise search architecture is the system behind a deceptively simple box. A user asks for a policy, customer decision, owner, incident, or precedent. The platform must discover evidence across many applications, preserve each source’s permissions, rank the right version, and return an answer that can be inspected and repaired.

The hard part is not choosing a vector database. It is designing a continuously reconciled information system: connectors, object models, content and ACL indexes, query planning, hybrid retrieval, ranking, citations, observability, and lifecycle controls all have to agree.

This guide explains a production reference architecture and the decisions that determine whether enterprise search remains trustworthy after the demo.

The short answer

A production enterprise search architecture has two connected paths:

1. The ingestion path discovers source objects, content, metadata, identities, relationships, and permissions; normalizes them; writes searchable indexes; and reconciles updates and deletions.

2. The query path authenticates the user, resolves their current principals, interprets the query, retrieves candidates, security-trims them, reranks authorized evidence, and returns results or a grounded answer with citations.

Around both paths sits an operational control plane for connector health, schema versions, freshness, ACL lag, evaluation, audit, and repair. If the control plane cannot prove that content and permissions are current, the platform should degrade safely rather than silently serve stale or overexposed results.

The reference architecture

The most useful decomposition is not “crawler plus AI.” It is seven cooperating layers.

1. Source and connector layer

Sources include document systems, chat, email, issue trackers, CRM, support platforms, code, data catalogs, intranets, and custom databases. Each connector must understand the source’s API, identity model, object types, hierarchy, sharing rules, update semantics, rate limits, and deletion behavior.

For every important source, define a connector contract:

• object types and fields covered

• content, comments, attachments, and tables parsed

• canonical source ID and deep-link URL

• users, groups, roles, and inherited permissions captured

• full crawl, incremental crawl, and webhook behavior

• update and deletion service-level objectives

• API throttling and backoff behavior

• unsupported objects and permission constructs

• reconciliation and replay procedure

Connector count is not architecture quality. A shallow connector may import titles but omit comments, tables, inherited ACLs, or deletion signals. That can make search look complete while failing the workflows that matter.

Glean’s connector documentation says a datasource crawl collects item content, metadata, and access permissions, then indexes the result into its knowledge graph. Its monitoring guidance separates crawling from indexing and exposes item count and change rate so teams can detect stalled synchronization. Glean: connect data sources Glean: connector monitoring

2. Ingestion and change-processing layer

Initial sync builds the corpus. Continuous change processing keeps it usable. A mature pipeline handles several event classes independently:

• object created, updated, moved, or deleted

• attachment or comment changed

• user or group created, changed, or deactivated

• membership added or removed

• sharing policy or inherited permission changed

• source schema or connector version changed

Do not assume content and permissions change together. A document may be unchanged while its folder becomes restricted. A person may leave the company without any document event. The architecture therefore needs separate content, identity, and permission checkpoints plus a reconciliation job that compares indexed state with the source of truth.

Use durable event IDs and idempotent writes. Store the source cursor, connector version, extraction version, object version, and last successful processing time. Dead-letter queues must preserve enough context to replay a failed object without recrawling the entire source.

For high-volatility systems, mix indexed and live retrieval. An index provides broad, fast recall; a query-time API fetch can supply very recent or structured data. Glean documents this pattern for Salesforce: scheduled indexing supports broad search, while eligible live-mode experiences retrieve fresh, permission-aware records through user-scoped OAuth. Glean: Salesforce connector

3. Parsing and canonical object layer

Search quality depends on what the index thinks an object is. The parser should produce a canonical envelope containing:

• tenant, source, connector, and immutable source object ID

• object type, title, body, sections, comments, and attachment references

• author, owner, timestamps, status, and source hierarchy

• canonical URL and precise citation targets

• language, entities, relationships, and business metadata

• content checksum and source version

• allow, deny, inherited, and visibility metadata

• lifecycle state: active, stale, deleted, quarantined, or failed

Chunking should preserve semantic and permission boundaries. A heading and its paragraph may belong together; a table row may require its column headers; a chat thread may need speaker and time context. Never merge chunks with different ACLs merely to improve semantic coherence.

Maintain parent-child relationships so a result can cite an exact passage while opening the canonical source object. Deduplicate copies without losing provenance. If three files repeat a policy, the system should know which is authoritative and which are derivatives.

Large or unsupported files require explicit behavior. Glean’s crawler limits documentation, for example, states that items over its content threshold may retain metadata and permissions even when body content is not downloaded. Buyers should test what becomes searchable and how the UI communicates partial indexing. Glean: crawler and indexing limits

4. Index and graph layer

No single index serves every enterprise query well. A practical architecture uses complementary structures:

Lexical index. An inverted index handles exact names, acronyms, ticket IDs, error strings, policy numbers, phrases, filters, and facets.

Vector index. Embeddings improve semantic recall for paraphrases, natural-language questions, and conceptually similar passages.

Metadata and structured index. Typed fields support dates, owners, source, status, geography, product, customer, and other exact constraints.

Permission index. Filterable principal and policy fields support security trimming. It must share stable object and chunk IDs with every retrieval index.

Relationship graph. Entities and edges connect people, teams, documents, customers, projects, products, events, and source authority. A graph can expand or rerank candidates, but traversal must remain permission-aware.

Store index versions and make rebuilds switchable. Blue-green indexing allows a new parser, embedding model, or schema to be evaluated before traffic moves. A rollback should restore the prior coherent set of content, vector, permission, and graph versions—not only one index.

Query execution from identity to answer

Step 1: authenticate and construct security context

The request enters with a trusted tenant and user identity. Resolve current direct and group principals, role claims, source-specific identities, and policy attributes. Do not accept a username supplied in the prompt.

The security context must travel through query planning, candidate retrieval, caches, reranking, answer generation, citations, traces, and follow-up queries. Cache keys need tenant and authorization scope; otherwise a correct result for one user can become a leak for another.

Step 2: understand and route the query

Query analysis identifies intent, entities, time constraints, requested object type, source preferences, and whether the user wants navigation, a list, a fact, or a synthesized answer.

Routing decisions may include:

• lexical search for an exact identifier

• vector search for a conceptual question

• structured filtering for dates, owners, or status

• graph expansion for relationships

• federated or live retrieval for volatile data

• clarification when the scope is ambiguous

Keep the query plan inspectable. When a result fails, operators should be able to see which routes ran, with what normalized terms and filters.

Step 3: retrieve and security-trim candidates

Apply authorization as early as the engine allows. Some systems filter candidates before scoring; others use native permission-aware retrieval; a fallback pattern stores user or group IDs as filterable fields and includes the caller’s principals in the query.

Microsoft documents this security-filter pattern for Azure AI Search: source documents carry principal IDs, those IDs are stored in a filterable index field, and the query excludes documents that do not match the requester’s user or group identifiers. It also warns that the principal string is a filtering input, not authentication by itself. Microsoft: security filters for Azure AI Search

Never retrieve sensitive text into a broadly visible model, reranker, trace, or cache and remove it only at presentation time. Presentation filtering is too late.

Step 4: fuse, rank, and diversify

Combine lexical, vector, structured, and graph candidates with a stable fusion method such as reciprocal rank fusion or a learned ranker. Then apply business and quality signals:

• authority and verification status

• freshness and effective date

• exactness and semantic relevance

• source quality

• user and team context

• object popularity or usage

• duplicate and near-duplicate suppression

• result diversity

Personalization may change ranking among authorized items; it must never change authorization.

Step 5: generate an answer from authorized evidence

For answer experiences, send only the selected authorized passages to the model. The answer contract should require:

• a direct response to the query

• citations for material claims

• visible uncertainty or conflicts

• source dates where recency matters

• abstention when evidence is insufficient

• no claim broader than the cited passages support

Citation IDs should bind to the exact evidence snapshot used for generation and also open the current canonical source. This lets operators reproduce the answer while users navigate to the live object.

Step 6: return results, citations, and trace

The response includes result cards or an answer, source links, snippets, and any warnings about stale or incomplete sources. The trace records query plan, index versions, principal-resolution outcome, candidate and filter counts, selected evidence, model version, latency, and cost.

Trace content is sensitive. Apply its own retention, redaction, and access policy.

Permission architecture and ACL propagation

Source systems express access differently: direct shares, groups, folder inheritance, domain links, record roles, project membership, row-level policies, public links, and explicit denies. Normalize enough for efficient evaluation without erasing source semantics.

A robust permission model stores:

• source principal IDs and mapped enterprise identities

• direct and inherited grants

• explicit denies where the source supports them

• group and nested-group relationships

• visibility classes such as private, organization, external, or public

• permission source, version, and observed time

• object hierarchy required for inheritance

Group expansion can happen during ingestion, at query time, or through a hybrid cache. Pre-expansion makes queries fast but revocation propagation harder. Query-time expansion is current but creates dependency and latency. Whichever model you choose, define a fail-closed response when identity or permission resolution is unavailable.

Permission freshness deserves its own service-level objective. “Index freshness: five minutes” is meaningless if ACL changes take a day. Track content lag, ACL lag, group-membership lag, and deletion lag separately.

Freshness, deletion, and reconciliation

Freshness is an end-to-end property, not a successful webhook. Measure:

source change → connector observation → parse → content index → ACL index → query visibility

Expose percentile lag by source and event type. Averages hide the restricted folder whose revocation failed.

Deletion must remove or tombstone every derivative: lexical document, chunks, embeddings, graph nodes and edges, caches, snippets, answer memories, and offline evaluation copies. Verify absence through the user-facing query path. Keep only the minimum audit tombstone required by policy.

Run periodic reconciliation even when webhooks exist. Compare source inventory and versions with indexed inventory. Sample ACL equivalence using test users. Repair drift through idempotent replay and record the outcome.

Reliability and observability

The control plane should answer four questions quickly:

1. Coverage: what should be searchable, and what actually is?

2. Freshness: how old are content, identity, permission, and deletion states?

3. Quality: which queries fail to retrieve or rank the right evidence?

4. Safety: could any user retrieve content they should not see?

Useful operational metrics include:

• objects discovered, processed, skipped, failed, and deleted

• source-to-index lag by percentile

• ACL and group-membership lag

• parser failure and partial-content rate

• lexical/vector/graph candidate contribution

• authorized candidate count before and after filtering

• zero-result and low-confidence rate

• citation correctness and answer groundedness

• query latency by stage

• cache hit rate by authorization scope

• connector throttling, retries, and dead letters

• index build, migration, and rollback status

Create a query debugger that can replay a request under a controlled test identity, show every stage, and explain why a result was included, excluded, or ranked. Restrict this tool carefully; an administrator should not gain blanket content access merely to debug relevance.

Evaluation before launch

Build an evaluation set from real missions and roles. Include exact lookup, exploratory search, cross-source questions, current-policy questions, relationship questions, and no-answer cases.

Measure retrieval separately from generation:

• candidate recall at K

• mean reciprocal rank or nDCG

• permission precision: unauthorized result rate must be zero

• citation correctness and coverage

• groundedness and completeness

• abstention quality

• freshness and deletion propagation

• latency and cost

Add adversarial permission tests: former employees, removed group members, restricted folders, externally shared items, nested groups, stale cached answers, and identical titles with different access. Test a source outage and identity-provider outage. A safe system should fail closed or clearly limit the experience.

Common architecture mistakes

Treating ingestion as a one-time migration

The corpus changes every minute. Without incremental events, reconciliation, and deletion proof, the index becomes a historical copy rather than enterprise search.

Storing content without source identity

If chunks lack immutable source IDs, versions, and canonical URLs, citations drift and deletion becomes unreliable.

Adding permissions after relevance

Post-filtering top results can leak content upstream and produce empty pages. Authorization belongs inside retrieval and every downstream data boundary.

Using one freshness metric

Content, ACLs, group membership, and deletion propagate through different paths. Track them independently.

Rebuilding only the vector index

Schema, parser, embedding, lexical, permission, and graph versions form a coherent release. Mixed versions create unexplained relevance and security behavior.

Hiding connector limitations

Unsupported object types, partial parsing, and source API gaps should appear in admin status and buyer documentation. Silent omissions destroy trust.

Evaluating fluent answers instead of retrieval

A model can produce a persuasive answer from weak evidence. Inspect candidate recall, authorization, ranking, citations, and abstention before judging prose.

A practical build sequence

Start with two or three high-value sources and five permission-sensitive missions. Define canonical object and ACL schemas before choosing indexes. Build deterministic ingestion, deletion, and reconciliation first. Add lexical and structured search, then vector retrieval and fusion. Add answer generation only after retrieval and citation evaluation are visible.

Run the system with test identities representing real roles. Establish content and permission freshness objectives. Add query tracing, replay, and repair. Only then expand connector coverage and autonomous actions.

The architecture should make failures observable and reversible. Search quality will evolve; permission correctness and provenance cannot be optional.

Final takeaway

Enterprise search is a synchronized, permission-aware evidence system. Connectors must capture content, metadata, identity, and ACLs. Indexes must support exact, semantic, structured, and relational retrieval. The query path must authenticate, security-trim, rank, cite, and trace. The control plane must measure freshness, reconcile drift, prove deletion, and support repair.

When those layers are designed together, AI answers become a reliable interface to company knowledge. When they are not, a polished search box merely hides stale data, missing evidence, and unsafe authorization assumptions.

Architecture control matrix

Frequently asked questions

What is the hardest part of enterprise search architecture?

Keeping content, identities, permissions, and deletions consistent across connectors and derivative indexes is usually harder than choosing a ranking model.

Should permissions be stored in the search index?

The retrieval path needs enough permission metadata or an authoritative query-time decision to exclude ineligible content before sensitive downstream processing.

How often should indexes be rebuilt?

Use incremental updates for normal operation and scheduled reconciliation or rebuilds for drift repair. The right frequency depends on source change rate and freshness SLOs.

What is the difference between search results and AI answers?

Search ranks items for a user to inspect. AI answers assemble and transform evidence, so they require stricter citation, context, abstention, and trace controls.

Which metrics prove the architecture works?

Track connector freshness, deletion latency, ACL correctness, recall, ranking quality, answer support, no-answer behavior, latency, and repair time.

Where Dokki fits

Dokki can be the governed destination and operating surface around enterprise search: canonical documents, structured records, decisions, and agent outputs remain reviewable in one workspace. A practical first step is to inventory one source domain in a Dokki table and attach freshness, permission, and owner evidence to every connector.

Sources

• Glean: Connect data sources

• Glean: Connector monitoring

• Glean: Crawler and indexing limits

• Microsoft: Security trimming for Azure AI Search

_Last verified: July 21, 2026._