ECS Task OOMKilled: Container Memory Limit Exceeded

If the problem involves database connection issues after container restarts, see our runbook on RDS connection limits. For general task stopping issues, see ECS Fargate Task stopped unexpectedly.

Symptoms

The task crashes and gets restarted by the ECS service scheduler:

# Check the reason the task was stopped
aws ecs describe-tasks \
  --cluster my-cluster \
  --tasks $(aws ecs list-tasks --cluster my-cluster --service-name my-service \
    --desired-status STOPPED --query 'taskArns[0]' --output text) \
  --query 'tasks[0].{Status:lastStatus,Reason:stoppedReason,Code:containers[0].exitCode}'

# Typical output:
# {
#   "Status": "STOPPED",
#   "Reason": "OutOfMemoryError: Container killed due to memory usage",
#   "Code": 137
# }

# Check memory metrics (last hour)
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name MemoryUtilization \
  --dimensions Name=ClusterName,Value=my-cluster Name=ServiceName,Value=my-service \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 --statistics Maximum \
  --query 'sort_by(Datapoints, &Timestamp)[-10:].[Timestamp,Maximum]'

Key indicators:

• Exit code 137 = SIGKILL (OOM killer)
• Exit code 1 with logs showing “java.lang.OutOfMemoryError” = JVM OOM (different problem)
• MemoryUtilization >95% before restart

Cause

1. Hard memory limit too low in task definition

The task definition sets memory (hard limit) to a value that the application regularly exceeds under load. The kernel’s OOM killer terminates the container.

2. Memory leak in the application

The application gradually allocates memory without releasing it (cache with no eviction policy, goroutine leak, unbounded queue). Usage grows linearly until it hits the limit.

3. JVM heap not properly constrained (Java <10)

A Java container with a 512 MB limit, but the JVM (older than Java 10 or Java 8 before 8u191) defaults to allocating 1/4 of host RAM, not the container limit. The JVM doesn’t see cgroup constraints and sets heap based on the physical machine memory.

Note: Java 10+ and Java 8u191+ respect cgroup limits by default (-XX:+UseContainerSupport is enabled by default). If you’re on a modern JVM, this issue shouldn’t occur unless you explicitly disabled UseContainerSupport.

4. Sidecar containers consuming shared memory

A task with an application container + sidecars (envoy, datadog-agent, log router). Their combined usage exceeds the task-level memory limit.

5. Traffic spike causes allocation burst

The application handles average traffic fine, but a sudden spike causes a one-time allocation burst (e.g., many concurrent responses buffered in memory).

Fix

A) Identify actual memory usage

# Container Insights (if enabled)
aws logs filter-log-events \
  --log-group-name "/aws/ecs/containerinsights/my-cluster/performance" \
  --filter-pattern '{ $.TaskId = "TASK_ID" }' \
  --start-time $(date -d '2 hours ago' +%s000)

# Alternatively - ECS Exec into a running container
aws ecs execute-command \
  --cluster my-cluster \
  --task RUNNING_TASK_ARN \
  --container my-app \
  --interactive \
  --command "/bin/sh"

# Inside the container:
cat /sys/fs/cgroup/memory/memory.usage_in_bytes
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
# Ratio usage/limit > 0.85 = close to OOM

B) Increase memory limit in task definition

{
  "family": "my-app",
  "cpu": "512",
  "memory": "1024",
  "containerDefinitions": [
    {
      "name": "my-app",
      "image": "my-app:latest",
      "memoryReservation": 768,
      "memory": 1024,
      "essential": true
    }
  ]
}

Rules:

• memory (hard limit) = maximum; the container is killed if it exceeds this value
• memoryReservation (soft limit) = value reserved during placement scheduling. The container can use more if the host has free memory (EC2 launch type only)
• On Fargate: memory at the task level is the hard limit for the entire task. The sum of all containers’ memoryReservation cannot exceed it.

# Register a new task definition revision
aws ecs register-task-definition --cli-input-json file://task-def.json

# Update the service
aws ecs update-service \
  --cluster my-cluster \
  --service my-service \
  --task-definition my-app:NEW_REVISION \
  --force-new-deployment

C) Right-sizing: determine the correct limit

# Collect usage data from the last 7 days (p99)
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name MemoryUtilization \
  --dimensions Name=ClusterName,Value=my-cluster Name=ServiceName,Value=my-service \
  --start-time $(date -u -v-7d +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 3600 --statistics p99 \
  --query 'max(Datapoints[].p99)'

Rule: hard limit = p99 usage * 1.3 (30% headroom). If p99 is 700 MB, set the limit to 910 MB (round up to the nearest Fargate value: 1024 MB).

Allowed values on Fargate:

D) Fix JVM memory (Java)

# Dockerfile - set JVM heap relative to container limit
ENV JAVA_OPTS="-XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0"

# OR explicitly:
# ENV JAVA_OPTS="-Xmx768m -Xms512m"
# Remember: JVM uses memory outside the heap (metaspace, threads, native)
# Rule: container memory = Xmx * 1.5

E) Constrain sidecar memory

{
  "containerDefinitions": [
    {
      "name": "my-app",
      "memory": 768,
      "memoryReservation": 512,
      "essential": true
    },
    {
      "name": "datadog-agent",
      "memory": 256,
      "memoryReservation": 128,
      "essential": false
    },
    {
      "name": "envoy",
      "memory": 128,
      "memoryReservation": 64,
      "essential": false
    }
  ],
  "memory": "1024"
}

The sum of container limits (768+256+128 = 1152) can exceed the task limit (1024) on EC2 because these are soft limits. On Fargate, the sum of container memory values must be <= task memory.

Verification

# Check if the new deployment runs stably
aws ecs describe-services \
  --cluster my-cluster --services my-service \
  --query 'services[0].{Running:runningCount,Desired:desiredCount,Deployments:deployments[0].status}'

# Monitor MemoryUtilization for 24h
# Should stay <80% of task memory after stabilization

# Check stop history (no new OOMs expected)
aws ecs list-tasks --cluster my-cluster --service-name my-service --desired-status STOPPED \
  --query 'taskArns' --output text | wc -w

If the service maintains runningCount == desiredCount for 24 hours with no exit code 137 restarts, the problem is resolved. Set a CloudWatch Alarm on MemoryUtilization >80% to react before the next OOM occurs.

Containers crashing with OOM?

Book a free 30-minute call. We'll help with right-sizing and stabilizing your ECS/Fargate services.

Book a call