Prometheus OOMKilled by High Cardinality: Diagnosis and Fix
Fix Prometheus OOMKilled: identify high cardinality metrics, configure metric_relabel_configs to drop unnecessary series, set series limits, and right-size memory allocation.
If you’re considering a move to managed observability (e.g., Datadog or Grafana Cloud) to avoid the headaches of self-hosted Prometheus, see our guide to migrating to Datadog. If you need to optimise observability costs, check out our article on the rising cost of monitoring.
Symptoms
The Prometheus pod keeps restarting with exit code 137 (OOMKilled):
# Check pod status
kubectl describe pod prometheus-server-0 -n monitoring | grep -A5 "Last State"
# Last State: Terminated
# Reason: OOMKilled
# Exit Code: 137
# Check current number of time series
curl -s http://prometheus:9090/api/v1/status/tsdb | jq '.data.headStats.numSeries'
# Check memory
kubectl top pod prometheus-server-0 -n monitoring
# Typical pattern: memory grows linearly until it hits the limit, then OOM
# How many series per target?
curl -s http://prometheus:9090/api/v1/targets | jq '[.data.activeTargets[].scrapePool] | group_by(.) | map({pool: .[0], count: length})'
Rule of thumb: Prometheus uses ~1-2 KB RAM per active time series. 1 million series = ~1.5-2 GB RAM. If you have 500K series with a 2 GB limit, you’re on the edge.
Cause
High cardinality means a metric generates too many unique label combinations. Typical sources of the explosion:
1. Dynamic labels with user/request IDs
# Each unique user_id creates a new series
http_requests_total{method="GET", path="/api/users", user_id="abc123"}
http_requests_total{method="GET", path="/api/users", user_id="def456"}
# ... x 100,000 users = 100K series from a single metric
2. Container/pod labels on short-lived workloads
# Each new pod creates a new series (pod_name contains a UUID)
container_cpu_usage_seconds_total{pod="worker-7f8b9c4d2-xk4rm", ...}
# After restart: new UUID = new series, the old one isn't removed immediately
3. Metrics with unbounded path labels
# Every unique URL is a new series
http_request_duration_seconds_bucket{path="/api/users/12345/orders/67890", le="0.5"}
# Combination of paths x buckets x methods = explosion
4. Misconfigured ServiceMonitor or PodMonitor
A ServiceMonitor scrapes an endpoint that exposes internal framework metrics with dynamic labels (e.g., gRPC per-method histograms with full paths).
Fix
A) Identify the highest-cardinality metrics
# Top 20 metrics by series count
curl -s http://prometheus:9090/api/v1/status/tsdb | jq -r '.data.seriesCountByMetricName | sort_by(.value) | reverse | .[0:20] | .[] | "\(.value)\t\(.name)"'
# Top 20 label pairs by series count
curl -s http://prometheus:9090/api/v1/status/tsdb | jq -r '.data.seriesCountByLabelValuePair | sort_by(.value) | reverse | .[0:20] | .[] | "\(.value)\t\(.name)"'
# Check a specific metric
curl -s 'http://prometheus:9090/api/v1/series?match[]=http_request_duration_seconds_bucket' | jq '.data | length'
B) Drop unnecessary metrics with metric_relabel_configs
# prometheus.yml or in ServiceMonitor spec
scrape_configs:
- job_name: 'my-app'
metric_relabel_configs:
# Drop high-cardinality metrics you don't need
- source_labels: [__name__]
regex: 'go_gc_.*|go_memstats_.*|process_.*'
action: drop
# Remove labels causing the explosion
- regex: 'user_id|session_id|request_id|trace_id'
action: labeldrop
# Limit path to known endpoints (replace the rest with "other")
- source_labels: [path]
regex: '/(api|health|metrics|ready)(.*)'
target_label: path
replacement: '/${1}'
- source_labels: [path]
regex: '/(?!api|health|metrics|ready).*'
target_label: path
replacement: '/other'
In Kubernetes with ServiceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app
spec:
endpoints:
- port: metrics
metricRelabelings:
- sourceLabels: [__name__]
regex: 'go_gc_.*|go_memstats_.*'
action: drop
- regex: 'user_id|request_id'
action: labeldrop
C) Set a per-target series limit
# prometheus.yml - global sample limit
scrape_configs:
- job_name: 'my-app'
sample_limit: 5000 # Reject scrape if the target returns >5000 series
# Alert in logs: "scrape failed: sample limit exceeded"
D) Right-size memory
# Calculate required memory:
# time_series * 2 KB = base usage
# + 30% overhead for queries and compaction
# + 20% safety buffer
# Example: 500K series
# 500,000 * 2 KB = 1 GB base
# + 30% = 1.3 GB
# + 20% = 1.56 GB -> set limit to 2 GB
# Kubernetes resource limit:
resources:
requests:
memory: 1.5Gi
limits:
memory: 2Gi
E) Consider remote write + shorter retention
If the series count is still too high even after cleanup, send data to external storage (Thanos, Cortex, Mimir) and shorten the local retention period:
# prometheus.yml
global:
scrape_interval: 30s # Consider increasing from 15s to 30s
storage:
tsdb:
retention.time: 6h # Short local retention
retention.size: 5GB # Or size-based limit
remote_write:
- url: http://mimir:9009/api/v1/push
Verification
# After changes - reload config (no restart needed!)
curl -X POST http://prometheus:9090/-/reload
# Wait 5 minutes, check series count
curl -s http://prometheus:9090/api/v1/status/tsdb | jq '.data.headStats.numSeries'
# Should drop within 2-3 scrape cycles
# Check memory
kubectl top pod prometheus-server-0 -n monitoring
# Verify alerts still work
curl -s http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules | length'
If the series count dropped by >30% and Prometheus hasn’t restarted for 24 hours, the problem is resolved. Set an alert on prometheus_tsdb_head_series with a threshold at 80% of your estimated maximum to catch regressions before they lead to OOM.
Prometheus crashing with OOM?
Book a free 30-minute call. We'll help you identify high cardinality sources and configure proper filtering.