Logo
AWS CloudWatch monitoring alerts

CloudWatch Alarm Not Triggering: Diagnosis and Fix

Fix a CloudWatch Alarm that won't trigger: INSUFFICIENT_DATA due to missing metrics, wrong evaluation periods, missing SNS permissions, and treat-missing-data configuration.

Jerzy Kopaczewski ·
You configured a CloudWatch Alarm for CPU >80%, but even though your instance is under full load, the alarm shows INSUFFICIENT_DATA or stays in OK. Or the alarm transitions to ALARM, but the notification never reaches Slack or email. This runbook covers both scenarios.

If you’re considering a move from CloudWatch to a more capable observability platform, see our guide to migrating to Datadog. To understand the CloudWatch cost model, see our article on hidden Lambda and EBS costs.

Symptoms

The alarm doesn’t react even though conditions are met:

# Check alarm state
aws cloudwatch describe-alarms \
  --alarm-names "my-cpu-alarm" \
  --query 'MetricAlarms[0].{State:StateValue,Reason:StateReason,Updated:StateUpdatedTimestamp}'

# Typical problematic responses:
# State: INSUFFICIENT_DATA
# Reason: "Unchecked: Initial alarm creation"
#
# or:
# State: OK
# Reason: "Threshold Crossing: 1 datapoint [45.2] was not greater than the threshold (80.0)"
# (but you know CPU is at 95%)

# Check if the metric has data at all
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Average

Cause

Scenario 1: INSUFFICIENT_DATA

The alarm is not receiving metric data. Possible reasons:

  • Detailed monitoring is not enabled. By default, EC2 reports metrics every 5 minutes. If your alarm period is 1 minute, it receives no data.
  • Wrong namespace or dimension. Typo in InstanceId, wrong Namespace.
  • Instance not reporting. CloudWatch agent isn’t running, or the instance is in a different region.
  • Custom metric not being published. The application stopped pushing the metric (crash, code bug).

Scenario 2: Alarm stays OK despite threshold breach

  • Evaluation period vs required datapoints. The alarm requires N out of M datapoints above the threshold. If you set 3 out of 3 and data arrives irregularly, the condition is never satisfied.
  • Average vs Maximum statistic. CPU spikes to 95%, but the average across a 5-minute window is 60%.
  • Treat missing data set to “missing.” Missing datapoints are treated as “no information,” not as a breach.

Scenario 3: Alarm triggered but no notification

  • SNS topic has no subscriptions. The topic exists, but nobody confirmed the subscription.
  • Empty alarm actions. AlarmActions doesn’t contain a topic ARN.
  • SNS permissions. A topic in a different account lacks the policy allowing publication.
  • Notification budget. SMS/email throttled by account limits.

Fix

A) INSUFFICIENT_DATA: fix the data source

# 1. Check if detailed monitoring is enabled (required for period <5min)
aws ec2 describe-instances \
  --instance-ids i-0123456789abcdef0 \
  --query 'Reservations[0].Instances[0].Monitoring.State'
# "disabled" = metrics every 5 minutes only

# Enable detailed monitoring (metrics every 1 minute)
aws ec2 monitor-instances --instance-ids i-0123456789abcdef0

# 2. Verify alarm dimensions vs actual data
aws cloudwatch describe-alarms --alarm-names "my-cpu-alarm" \
  --query 'MetricAlarms[0].Dimensions'
# Make sure InstanceId is correct and free of typos

# 3. If it's a custom metric - check if the app is publishing it
aws cloudwatch list-metrics \
  --namespace "MyApp" \
  --metric-name "RequestLatency"
# Empty result = metric is not being sent

B) Fix evaluation period and statistic

# Change alarm to use Maximum instead of Average (for spike detection)
aws cloudwatch put-metric-alarm \
  --alarm-name "my-cpu-alarm" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Maximum \
  --period 60 \
  --evaluation-periods 3 \
  --datapoints-to-alarm 2 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --alarm-actions arn:aws:sns:eu-central-1:123456789012:ops-alerts \
  --treat-missing-data breaching
  # ^^ breaching = missing data treated as threshold breach
  # Alternatives: missing (ignore), notBreaching (treat as OK), ignore

Key parameters:

  • --period 60 - window in seconds (must be >= metric reporting frequency)
  • --evaluation-periods 3 - how many recent periods to check
  • --datapoints-to-alarm 2 - how many of the evaluation-periods must breach the threshold (M of N)
  • --treat-missing-data breaching - critical for custom metrics that might stop arriving

C) Fix notifications (SNS)

# 1. Check if the alarm has actions configured
aws cloudwatch describe-alarms --alarm-names "my-cpu-alarm" \
  --query 'MetricAlarms[0].AlarmActions'
# Should return an SNS topic ARN. If empty - add one.

# 2. Check topic subscriptions
aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:eu-central-1:123456789012:ops-alerts
# Each subscription must have a SubscriptionArn (not "PendingConfirmation")

# 3. If the topic is in another account - add a policy
aws sns set-topic-attributes \
  --topic-arn arn:aws:sns:eu-central-1:123456789012:ops-alerts \
  --attribute-name Policy \
  --attribute-value '{
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "cloudwatch.amazonaws.com"},
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:eu-central-1:123456789012:ops-alerts"
    }]
  }'

# 4. Force a test notification
aws cloudwatch set-alarm-state \
  --alarm-name "my-cpu-alarm" \
  --state-value ALARM \
  --state-reason "Manual test"
# Check if the notification arrived. Reset to OK after the test:
aws cloudwatch set-alarm-state \
  --alarm-name "my-cpu-alarm" \
  --state-value OK \
  --state-reason "Reset after test"

Verification

# Check current alarm state
aws cloudwatch describe-alarms --alarm-names "my-cpu-alarm" \
  --query 'MetricAlarms[0].{State:StateValue,Reason:StateReason}'

# Check state transition history
aws cloudwatch describe-alarm-history \
  --alarm-name "my-cpu-alarm" \
  --history-item-type StateUpdate \
  --max-records 5

# Force threshold breach (stress test)
# On the EC2 instance:
stress-ng --cpu 4 --timeout 300s
# Alarm should transition to ALARM within evaluation-periods * period seconds

If the alarm transitions to ALARM in response to real load and the notification reaches its recipients, the problem is resolved.

CloudWatch Alarms most commonly fail for three reasons: missing data (wrong period or disabled detailed monitoring), an overly strict condition (3/3 datapoints with Average), or a broken notification chain (SNS without confirmed subscriptions). The treat-missing-data parameter is critical for custom metrics because the default value of "missing" means the alarm won't react when the application stops sending data entirely.
Jerzy Kopaczewski

Alerting not working as expected?

Book a free 30-minute call. We'll review your monitoring configuration and fix dead alerts.