Datadog Agent Consuming Too Much CPU or Memory: How to Diagnose and Fix
Diagnose and fix high CPU/memory usage by the Datadog Agent: disable unnecessary integrations, limit custom metrics, configure log collection, and switch to Cluster Agent.
If you’re planning a migration to Datadog from another platform, see our guide to migrating to Datadog. If you need help with Datadog configuration and optimization, check out our Datadog consulting service.
Symptoms
The agent is consuming significantly more resources than expected:
# Check agent resource usage
# On Linux:
top -p $(pgrep -f "datadog-agent run")
# On Kubernetes (DaemonSet):
kubectl top pods -n datadog -l app=datadog-agent
# Expected values (typical host with 10-15 integrations):
# CPU: 0.5-2% (single core)
# RAM: 150-300 MB
# If you see >5% CPU or >500 MB RAM - you have a problem
Check the agent status to see what’s running:
# Agent status with integration details
sudo datadog-agent status
# Check how many integrations are active
sudo datadog-agent status | grep -c "Instance ID"
# Check how many custom metrics are being reported
sudo datadog-agent status | grep "Total number of custom metrics"
Cause
Five most common reasons for high resource consumption:
1. Too many active integrations
Each integration (check) runs every 15-30 seconds. With 30+ integrations on a single host, the agent spends a significant portion of its time polling services.
2. High cardinality metrics (DogStatsD)
The application sends metrics with dynamic tags (e.g., user_id, request_id, session_token). Each unique combination of metric name + tags is a separate time series that the agent must buffer and send.
3. Tailing large log files
The agent is tailing a high-throughput log file (>1000 lines/s). Parsing and sending each line requires CPU.
4. Process monitoring with full arguments
process_config.enabled: true with process_config.scrub_args: false collects full arguments of every process. On a host running hundreds of processes, this creates significant overhead.
5. APM with high trace volume
The agent collects and processes traces from multiple services on a single host. At >1000 spans/s, the agent needs more resources for sampling and aggregation.
Fix
A) Identify what’s consuming resources
# Per-integration resource profile
sudo datadog-agent diagnose show-metadata agent-checks
# Check execution time of each check
sudo datadog-agent status | grep -A2 "Last Execution"
# On Kubernetes - check agent logs for slow operations
kubectl logs -n datadog -l app=datadog-agent --tail=100 | grep -i "slow\|timeout\|warning"
B) Disable unnecessary integrations
# /etc/datadog-agent/conf.d/ - remove or move .yaml files for integrations
# you don't need
# Commonly unnecessary integrations on application servers:
# - disk (enabled by default, often not needed on containers)
# - network (enabled by default, consider disabling on short-lived containers)
# - ntp (rarely needed when host has chronyd)
# Disable an integration without deleting the file:
# Rename: mv conf.d/ntp.d/conf.yaml conf.d/ntp.d/conf.yaml.disabled
C) Limit custom metrics (DogStatsD)
# /etc/datadog-agent/datadog.yaml
# Limit the number of unique tags per metric
dogstatsd_mapper_profiles:
- name: reduce_cardinality
prefix: "app."
mappings:
- match: "app.request.*"
name: "app.request"
tags:
endpoint: "$1"
# DO NOT add user_id, session_id, etc.
# Alternatively - set a hard limit
dogstatsd_metrics_stats_enable: true
# Monitor in Datadog: datadog.dogstatsd.metrics.unique_metrics
In your application code, never use dynamic values (user ID, timestamp, UUID) as tags:
# BAD - creates millions of unique time series
statsd.increment('api.request', tags=[f'user:{user_id}'])
# GOOD - bounded cardinality
statsd.increment('api.request', tags=[f'endpoint:{endpoint}', f'status:{status_code}'])
D) Optimise log collection
# /etc/datadog-agent/conf.d/app_logs.d/conf.yaml
logs:
- type: file
path: /var/log/app/production.log
service: my-app
source: python
# Limit throughput
# Maximum lines per second from this file
# (default: no limit)
# processing_rules:
# - type: throttle
# limit: 500 # max 500 lines/s
# Exclude unnecessary lines (health checks, debug)
processing_rules:
- type: exclude_at_match
name: exclude_health_checks
pattern: "GET /health"
- type: exclude_at_match
name: exclude_debug
pattern: "DEBUG"
E) Switch to Cluster Agent (Kubernetes)
On Kubernetes clusters with >10 nodes, the Cluster Agent centralizes operations that don’t need to run per-node:
# values.yaml (Helm chart datadog/datadog)
clusterAgent:
enabled: true
replicas: 2
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
agents:
# Reduce per-node agent resources after enabling Cluster Agent
containers:
agent:
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 300m
memory: 512Mi
F) Set resource limits (Kubernetes)
# Prevent the "agent is stealing app resources" situation
# DaemonSet spec:
containers:
- name: agent
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m # Agent will be throttled above this
memory: 512Mi # Agent will be OOMKilled above this
Verification
# After changes - restart the agent
sudo systemctl restart datadog-agent
# Wait 2-3 minutes, check the new resource usage
top -p $(pgrep -f "datadog-agent run") -n 3 -b
# Check if data is still flowing to Datadog
sudo datadog-agent status | grep "Running Checks"
sudo datadog-agent status | grep "Total Metrics"
# On Kubernetes
kubectl top pods -n datadog -l app=datadog-agent
If CPU dropped below 5% and RAM below 400 MB, the problem is resolved. Monitor the datadog.agent.running and system.cpu.user metrics with a process:datadog-agent filter in Datadog to catch regressions.
Datadog consuming too many resources?
Book a free 30-minute call. We'll review your agent configuration and optimise resource usage.