Production RAG: Beyond a Vector Database

The system needs governed ingestion, reliable retrieval, source citations, access control, stale-content handling, observability, evals, and integration with business systems.

Payments examples show why this matters. A system answering questions about instant payouts, open banking, refunds, customer data, and agentic commerce must be accurate, current, permission-aware, and auditable.

The Required Architecture

Production RAG has two main flows.

Knowledge preparation

approved sources
  -> classification
  -> PII scanning
  -> metadata enrichment
  -> chunking
  -> embeddings
  -> vector/search index

Runtime answering

user question
  -> identity and policy check
  -> retrieval
  -> evidence filtering
  -> model response
  -> citation validation
  -> trace and eval data

The first flow prepares trusted knowledge. The second flow uses that knowledge safely when a user asks a question.

What Should Go Into the Knowledge Base

Good candidates include:

• product policies
• operational runbooks
• approved support procedures
• compliance-reviewed guidance
• architecture documents
• incident postmortems approved for reuse
• API integration guides

Payments examples include:

• instant payout investigation runbook
• open banking account funding policy
• agent wallet control policy
• customer data handling policy
• settlement and reconciliation guide

Bad candidates for a general RAG index include:

• raw customer contact records
• payment credentials
• live transaction records
• refund case history
• account-level open banking data

Customer-specific and live data should come from authenticated APIs and tools, with row-level access control and audit logs.

Why Metadata Matters

Every chunk should carry metadata.

{
  "classification": "internal",
  "allowedGroups": ["payments-ops", "sre"],
  "productArea": "instant-payouts",
  "containsPii": false,
  "freshness": "current",
  "owner": "payments-ops"
}

Metadata allows the system to answer questions like:

• Is this user allowed to see this evidence?
• Is the document current?
• Does it contain PII?
• Which product area owns it?
• Which region or tenant does it apply to?

This is how governance becomes executable in the system.

Access Control

Access control should happen before the model sees evidence.

If a deployment rollback runbook is restricted to SRE users, the retrieval layer should return it only for authorized SRE users.

Weak approach:

Retrieve restricted evidence and tell the model not to reveal it.

Required approach:

Filter restricted evidence before it enters model context.

This protects sensitive operational knowledge and reduces reliance on prompt instructions.

PII and Customer Data

RAG should not become an uncontrolled copy of customer data.

For example, a customer refund status question should not be answered from a vector index full of customer records.

Required pattern:

Policy question -> RAG
Customer-specific question -> authenticated API/tool

Example:

What is the refund policy for Brazil?
  -> RAG can answer from approved policy.

What is the status of customer refund case 123?
  -> API/tool should answer from the system of record.

This distinction is important for privacy, compliance, and operational accuracy.

For a deeper treatment of this boundary, read What Customer Data Belongs in RAG?.

Chunking and Embeddings

Chunking controls the evidence boundary. A chunk should contain enough context to answer a question without mixing unrelated topics.

Bad chunk:

Check payout status.

Better chunk:

When investigating a delayed instant payout, check payout status, rail selection,
issuer response, tokenization status, risk decision, and downstream processor health.

Embeddings turn chunks and queries into vectors. Vector search then retrieves chunks with similar meaning.

Common technology choices include:

• Azure AI Search
• Amazon OpenSearch
• Amazon S3 Vectors
• pgvector
• Pinecone
• Azure OpenAI embeddings
• Amazon Bedrock embeddings

For enterprise use, hybrid retrieval is often useful because it combines semantic matching with exact keyword matching.

Evidence Coverage

The system should check whether retrieved evidence actually covers the question.

Question: What does the knowledge base say about customer support refunds in Brazil?

A customer data handling policy may be related to the word "customer," but it does not answer refunds in Brazil. The right response should say that specific evidence is missing, rather than turning a related policy into a direct answer.

This is a common source of RAG errors: the evidence is related, but the answer is unsupported.

Citations

A production RAG answer should cite exact sources.

Weak:

Source: Instant Payouts Runbook

Better:

Source: payments-instant-payouts-runbook.md:15-31

Exact citations help with:

• user trust
• audit review
• compliance review
• support escalation
• engineering debugging

The application should validate citations after the model generates the answer.

Repair and Fallback

If the answer is missing exact citations, the system can ask the model to repair the answer using only allowed source strings. If repair fails, the system should have deterministic fallback behavior.

Relevant evidence was found, but the generated answer did not include valid exact
citations. Available source:
- payments-instant-payouts-runbook.md:15-31

This is less conversational, but it is safer and easier to audit.

Observability

A production RAG trace should show:

• what the user asked
• which user groups and policies applied
• which chunks were retrieved
• why chunks were selected
• which chunks were filtered
• whether evidence covered the question
• whether citations passed validation
• whether repair or fallback was triggered
• latency by stage
• cost by stage

This allows teams to debug quality, reliability, latency, and governance issues.

Evals

RAG systems need evals before launch and during ongoing improvement.

Retrieval evals:

Did the system retrieve the expected evidence?

Answer evals:

Did the final answer use evidence correctly and cite it?

Useful payments eval cases include:

• delayed instant payout triage
• delegated authority and spending limits
• open banking account funding
• customer data handling
• refund policy by market
• restricted SRE runbook
• stale legacy runbook
• unsupported question

Negative controls should be included so the team knows the eval suite can catch bad behavior.

What Buyers Should Ask Vendors or Internal Teams

Before approving a production RAG system, ask:

• Which sources are indexed?
• Who owns the content?
• How is PII handled?
• How are documents classified?
• How are permissions enforced?
• Where are chunks and embeddings stored?
• Which embedding model is used?
• Which vector or search backend is used?
• Is retrieval hybrid or vector-only?
• How are stale documents excluded?
• Are citations exact?
• Are citations validated?
• What happens when validation fails?
• What traces are available?
• What evals are run?
• How are live customer-specific questions handled?

These questions separate a prototype from a system that can be trusted in production.

DIS AI Perspective

DIS AI focuses on implementation-first AI systems: practical architecture, working integrations, measurable quality, and controls that fit real business workflows.

For RAG, that means designing the full path:

• knowledge ingestion
• metadata and governance
• vector and hybrid retrieval
• access control
• citation validation
• observability
• evals
• tool and API integration

This is especially important in domains such as payments, banking, customer support, operations, and agentic commerce, where the answer must be accurate, current, permission-aware, and traceable.