OpenSearch Serverless search latency spikes: diagnosing OCU starvation in vector search workloads

Symptoms

Your RAG system’s retrieval step degrades under load:

• k-NN vector search queries slow from 200ms (p50) to 2-5 seconds (p50)
• No errors in CloudWatch - requests complete, just slowly
• Degradation correlates with traffic spikes (peak hours, batch ingestion)
• OpenSearch Serverless dashboard shows SearchOCU utilisation at 90-100%
• End-to-end RAG latency exceeds acceptable thresholds (>5s total)

CloudWatch metrics to check:

Namespace: AWS/AOSS
Metric: SearchOCU (Average)
Expected: < 70% for healthy operation
Actual: 90-100% sustained

Cause

OpenSearch Serverless allocates a fixed number of Search OCUs to your collection. Each OCU provides a unit of compute for query execution. When all OCUs are busy:

1. New queries enter an internal queue
2. Queue depth increases proportionally to traffic
3. Latency rises linearly - no circuit breaker, no errors, just slowness

Common triggers:

• Traffic growth - your RAG queries doubled but OCUs stayed at the minimum (2 search OCUs)
• Large k values - using k=20 or higher makes each query compute-intensive
• Missing pre-filters - every query searches the entire vector index instead of a filtered subset
• High ef_search - default ef_search=512 is expensive; most workloads work fine with 100-200
• Concurrent batch ingestion - indexing OCUs at capacity can indirectly affect search performance on shared resources

Fix

Step 1: Confirm OCU saturation

Check the CloudWatch dashboard for your collection:

aws cloudwatch get-metric-statistics \
  --namespace AWS/AOSS \
  --metric-name SearchOCU \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Average \
  --region eu-central-1

If average is consistently above 80%, you need more capacity or query optimisation.

Step 2: Reduce query cost (immediate relief)

Reduce k value:

{
  "size": 5,
  "query": {
    "knn": {
      "embedding": {
        "vector": [0.1, 0.2, ...],
        "k": 5
      }
    }
  }
}

Changing from k=20 to k=5 reduces compute per query by ~60%. For most RAG use cases, 5 chunks provide sufficient context.

Add metadata pre-filters:

{
  "size": 5,
  "query": {
    "knn": {
      "embedding": {
        "vector": [0.1, 0.2, ...],
        "k": 5,
        "filter": {
          "bool": {
            "must": [
              {"term": {"tenant_id": "customer-123"}},
              {"term": {"doc_type": "policy"}}
            ]
          }
        }
      }
    }
  }
}

Pre-filters reduce the search space before k-NN computation. If you have 1M vectors but the filter narrows to 10K, each query is 100x cheaper.

Lower ef_search (index setting):

{
  "settings": {
    "index": {
      "knn.algo_param.ef_search": 100
    }
  }
}

Default 512 provides 99.5%+ recall. Dropping to 100 gives ~98% recall at 3-5x lower compute cost. Measure against your test set.

Step 3: Increase OCU capacity

OpenSearch Serverless auto-scales OCUs within your configured limits, but the maximum may be too low. Contact AWS support or adjust via the console:

1. Go to OpenSearch Serverless > Collections > your collection
2. Check “Capacity” settings
3. Increase maximum search OCUs (e.g. from 2 to 4 or 8)

Note: OCU increases take effect within minutes. Each additional search OCU costs ~$0.24/hour (~£140/month).

Step 4: Evaluate provisioned OpenSearch (for sustained high load)

If you consistently need >4 search OCUs, a provisioned OpenSearch domain with k-NN plugin may be more cost-effective:

• Provisioned: you control instance size and count, autoscaling via CloudWatch alarms
• Typical: 2x r6g.large.search instances with k-NN = ~$350/month with predictable performance
• Trade-off: you manage the cluster (updates, shards, snapshots) but gain full control over capacity

Validation

After applying fixes, confirm latency returns to baseline:

# Query your collection and measure response time
time curl -s -X POST "https://your-collection.eu-central-1.aoss.amazonaws.com/your-index/_search" \
  -H "Content-Type: application/json" \
  --aws-sigv4 "aws:amz:eu-central-1:aoss" \
  -d '{"size":5,"query":{"knn":{"embedding":{"vector":[0.1,0.2],"k":5}}}}'

Expected: response time < 300ms.

Monitor CloudWatch:

• SearchOCU average < 70%
• SearchLatency p99 < 500ms

Related

• RAG Architecture on AWS: S3 + OpenSearch + Bedrock - full architecture guide including OCU cost planning
• OpenSearch Serverless capacity and scaling documentation