RAG Architecture on AWS - S3 + OpenSearch + Bedrock

What is RAG and why infrastructure matters more than the model

Retrieval-Augmented Generation is an architecture pattern where an LLM generates answers using context retrieved from an external knowledge base, rather than relying solely on its training data. The system retrieves relevant document chunks at query time and injects them into the prompt as context.

From an infrastructure perspective, RAG has three distinct layers that each need to scale independently:

1. Ingestion layer - documents flow in, get chunked, embedded, and indexed
2. Retrieval layer - user queries are embedded and matched against stored vectors
3. Generation layer - retrieved context is combined with the query and sent to an LLM

The model choice (Bedrock vs SageMaker vs self-hosted) is covered in our AWS LLM inference comparison. This article focuses on layers 1 and 2 - the infrastructure that determines whether RAG works reliably at scale.

“Just call the API” breaks when:

• Your document corpus exceeds 10,000 pages and ingestion takes hours
• Query latency spikes because your vector index is not provisioned correctly
• Costs escalate because you are embedding documents repeatedly instead of caching
• Retrieval quality degrades because chunking strategy was an afterthought

Reference architecture - S3 + OpenSearch Serverless + Bedrock

Data flow

Documents (PDF, HTML, Markdown)
    |
    v
S3 Bucket (raw storage)
    |
    v
Lambda / Step Functions (chunking + preprocessing)
    |
    v
Bedrock Embeddings API (Titan Embeddings v2)
    |
    v
OpenSearch Serverless (vector index, k-NN search)
    |
    v
Query: embed user question -> k-NN search -> retrieve top-k chunks
    |
    v
Bedrock Generation (Claude Sonnet 4.6 / Haiku) with retrieved context
    |
    v
Response to user

Component breakdown

• S3 - document storage with versioning and lifecycle policies. Source of truth for all raw content.
• Lambda / Step Functions - ingestion pipeline orchestration. Handles chunking, metadata extraction, and calls to Bedrock Embeddings.
• Bedrock Titan Embeddings v2 - converts text chunks into 1024-dimensional vectors. Available in-region in eu-west-2 and eu-central-1.
• OpenSearch Serverless (vector engine) - stores vectors and performs approximate k-nearest-neighbour (k-NN) search at query time.
• Bedrock generation model - takes the user query + retrieved chunks and generates an answer. Claude Sonnet 4.6 for quality, Haiku 3.5 for speed/cost.

Alternative - Bedrock Knowledge Bases (fully managed RAG)

AWS offers Bedrock Knowledge Bases as a fully managed RAG solution. It handles chunking, embedding, indexing, and retrieval automatically.

How it works

1. You point Knowledge Bases at an S3 bucket containing your documents
2. AWS automatically chunks, embeds (using Titan Embeddings), and indexes them in a managed vector store
3. At query time, you call a single API that retrieves context and generates a response

When to use Knowledge Bases

• Prototyping RAG quickly (minutes to first query)
• Document corpus under 50,000 pages
• Default chunking strategy (fixed-size with overlap) is acceptable
• You do not need custom retrieval logic or re-ranking

When to build your own

• You need custom chunking (semantic, hierarchical, or domain-specific)
• You require hybrid search (vector + keyword BM25)
• Document corpus exceeds 100K pages and you need control over indexing throughput
• You want to use a different vector store (Pinecone, pgvector, Qdrant)
• You need multi-tenancy with strict data isolation

Cost comparison

Knowledge Bases pricing: storage per GB/month + queries per request. For small corpora, it can be cheaper than self-managed OpenSearch. At scale, the per-query pricing adds up and self-managed becomes more economical.

OpenSearch Serverless - pricing, scaling, and cost traps

OpenSearch Serverless is the default vector store for custom RAG on AWS. It abstracts away cluster management but introduces its own complexity through the OCU (OpenSearch Compute Unit) model.

OCU model explained

OpenSearch Serverless charges in OCUs - each OCU provides a unit of compute for either indexing or search:

• Indexing OCUs - handle document ingestion and vector indexing
• Search OCUs - handle query execution and k-NN search

The minimum is 2 OCUs for search and 2 OCUs for indexing (4 OCUs total). Each OCU costs approximately $0.24/hour in eu-central-1.

Common cost traps

• Minimum 4 OCUs always running - even with zero traffic, you pay ~£560/month. There is no scale-to-zero.
• Collection types matter - use “VECTORSEARCH” collection type, not “SEARCH” or “TIMESERIES”. Wrong type wastes OCUs on capabilities you do not need.
• Indexing OCUs during bursts - large batch ingestion can temporarily require more indexing OCUs. Schedule ingestion during off-peak hours.

When to use provisioned OpenSearch instead

If your workload is large and predictable (>1M documents, stable query volume), a provisioned OpenSearch domain with k-NN plugin can be cheaper than Serverless. You trade operational simplicity for cost control.

Embedding pipeline - batch vs real-time

Batch ingestion (Step Functions + Lambda)

For initial document loading and periodic updates:

# Lambda function: chunk document and generate embeddings via Bedrock
import boto3
import json

bedrock = boto3.client("bedrock-runtime", region_name="eu-central-1")
s3 = boto3.client("s3")

def lambda_handler(event, context):
    bucket = event["bucket"]
    key = event["key"]

    # Fetch document from S3
    response = s3.get_object(Bucket=bucket, Key=key)
    text = response["Body"].read().decode("utf-8")

    # Chunk the document (simple fixed-size with overlap)
    chunks = chunk_text(text, chunk_size=512, overlap=64)

    # Generate embeddings for each chunk
    embeddings = []
    for chunk in chunks:
        embed_response = bedrock.invoke_model(
            modelId="amazon.titan-embed-text-v2:0",
            body=json.dumps({"inputText": chunk}),
        )
        vector = json.loads(embed_response["body"].read())["embedding"]
        embeddings.append({"text": chunk, "vector": vector})

    return {"chunks_processed": len(embeddings), "embeddings": embeddings}


def chunk_text(text, chunk_size=512, overlap=64):
    """Split text into overlapping chunks of roughly chunk_size tokens."""
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = " ".join(words[start:end])
        chunks.append(chunk)
        start = end - overlap
    return chunks

Real-time ingestion (API Gateway + Lambda)

For user-uploaded documents that need to be searchable immediately:

• API Gateway receives the upload
• Lambda chunks and embeds in real-time
• Vectors are indexed in OpenSearch immediately
• Latency: 2-10 seconds depending on document size

Embedding costs

Bedrock Titan Embeddings v2 costs $0.00002 per 1K input tokens (one of the cheapest embedding options available).

Example: embedding 10,000 documents averaging 2,000 tokens each:

• 10,000 x 2,000 / 1,000 x $0.00002 = $0.40 total

Embedding costs are negligible. The real cost is OpenSearch OCUs and generation tokens.

Scaling patterns

10K documents vs 10M documents

At 10K documents, everything fits in the minimum OpenSearch configuration (2 search OCUs). At 10M documents, you need to think about:

• Index sharding - distribute vectors across multiple shards for parallel search
• Tiered ingestion - process documents in priority order; not everything needs immediate indexing
• Metadata filtering - use OpenSearch pre-filters to reduce the k-NN search space before vector similarity
• Caching - cache frequent query embeddings and retrieval results in ElastiCache/DynamoDB

Query latency optimisation

• Set ef_search parameter based on your precision/speed trade-off (higher = more accurate, slower)
• Use approximate k-NN (HNSW algorithm) rather than exact search
• Limit k to 5-10 chunks - more context does not always improve generation quality
• Pre-compute embeddings for common queries

Cost breakdown for three scenarios

Key insight: at small/medium scale, OpenSearch Serverless dominates the bill. At large scale, Bedrock generation tokens take over. Optimise accordingly - at low volume, consider Knowledge Bases to avoid the OCU minimum; at high volume, optimise prompt length and caching.

Security and data governance

Network isolation

• Deploy OpenSearch Serverless with VPC access (no public endpoint)
• Use Bedrock VPC endpoints so API calls never traverse the public internet
• Place Lambda functions in VPC subnets with appropriate security groups

Scotland’s AI Strategy 2026-2031 places explicit emphasis on responsible AI deployment and data governance - making VPC isolation and audit trails a regulatory expectation for organisations building RAG systems that process sensitive data.

IAM least privilege

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel"
      ],
      "Resource": [
        "arn:aws:bedrock:eu-central-1::foundation-model/amazon.titan-embed-text-v2:0",
        "arn:aws:bedrock:eu-central-1::foundation-model/anthropic.claude-3-5-haiku-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "aoss:APIAccessAll"
      ],
      "Resource": "arn:aws:aoss:eu-central-1:*:collection/*"
    }
  ]
}

Encryption and audit

• OpenSearch Serverless encrypts data at rest by default (AWS-managed keys or CMK)
• Enable CloudTrail logging for all Bedrock and OpenSearch API calls
• S3 server-side encryption for all documents
• No data retention in Bedrock - prompts and responses are not stored or used for training

Common mistakes and how to avoid them

Chunking too large or too small

• Too large (>1000 tokens): retrieved chunks contain too much irrelevant context, diluting the signal
• Too small (<100 tokens): chunks lose coherence and the LLM cannot reason over fragments
• Sweet spot: 256-512 tokens with 10-15% overlap for most document types
• Exception: code, tables, and structured data benefit from semantic chunking (split on logical boundaries)

Not tuning k-NN parameters

• Default ef_search=512 is often overkill for production workloads
• Start with ef_search=100 and measure recall against a test set
• Use k=5 as default; increasing to 10+ rarely improves answer quality but doubles generation cost

Ignoring embedding model choice

• Titan Embeddings v2 (1024 dimensions) is a solid default for English content
• For multilingual content, consider Cohere Embed v3 (available on Bedrock)
• Embedding dimension directly affects OpenSearch storage costs and search speed

Skipping retrieval evaluation

• Build a test set of 50-100 question/answer pairs from your actual documents
• Measure retrieval precision (are the right chunks being returned?) before optimising generation
• Tools: RAGAS framework, custom precision@k metrics
• Without evaluation, you are guessing whether poor answers stem from retrieval or generation

Frequently asked questions

Is OpenSearch Serverless the only vector store option on AWS?

No. Alternatives include: Amazon Aurora with pgvector extension (cheaper for small workloads, no OCU minimum), Amazon MemoryDB (for ultra-low-latency retrieval), Amazon Neptune Analytics (for graph-based RAG), or external services like Pinecone. OpenSearch Serverless is the default choice for its native integration with Bedrock Knowledge Bases.

How much does a production RAG system cost on AWS?

For a team processing 50,000 queries/day against 500K documents using Claude 3.5 Haiku: approximately £2,000-£2,500/month. The two biggest cost items are OpenSearch Serverless OCUs (~£840) and Bedrock generation tokens (~£1,500). Embedding costs are negligible.

Can I use RAG with models in eu-central-1 for GDPR compliance?

Yes. Titan Embeddings v2 and Claude Sonnet 4.6 are available in-region in eu-central-1 (Frankfurt). For newer models like Claude Sonnet 5 or Opus 4.8, EU Geographic routing keeps data within EU boundaries. OpenSearch Serverless in eu-central-1 ensures your vector index stays in Germany.

What is the latency for a RAG query end-to-end?

Typical end-to-end latency for a RAG query (embed question + k-NN search + generate answer): 1.5-3 seconds with Claude Sonnet 4.6, 800ms-1.5s with Claude 3.5 Haiku. The retrieval step (embed + search) takes 100-300ms; generation dominates total latency.

Should I use Bedrock Knowledge Bases or build my own?

Start with Knowledge Bases if: your corpus is under 50K pages, default chunking works, and you want to ship in days not weeks. Build your own if: you need custom chunking, hybrid search, multi-tenancy, or cost control over OpenSearch OCUs at scale. Many teams prototype with Knowledge Bases then migrate to self-managed as requirements solidify.

Summary

• RAG on AWS is a three-layer architecture: ingestion (S3 + Lambda + Embeddings), retrieval (OpenSearch Serverless), and generation (Bedrock). Each layer has distinct scaling characteristics and cost drivers.
• OpenSearch Serverless is the primary cost driver at low-medium volume (minimum ~£560/month for 4 OCUs). At high volume, Bedrock generation tokens dominate.
• Bedrock Knowledge Bases offer the fastest path to production RAG but sacrifice control over chunking, retrieval tuning, and cost optimisation.
• Chunking strategy has more impact on answer quality than model choice. Invest in evaluation (RAGAS, precision@k) before optimising the LLM.
• For EU data residency, deploy OpenSearch Serverless and Bedrock in eu-central-1 (Frankfurt). All core RAG components are available in-region.
• NHS Digital teams in Leeds running clinical decision support and fintech companies across Manchester’s AI corridor face the same architecture decisions - the infrastructure patterns are universal, but compliance requirements (FCA, ICO, GDPR) shape the security layer.

Need help designing or optimising your RAG architecture on AWS? The Devopsity engineering team builds production retrieval systems for organisations across the UK and EU.