Logo
Kubernetes EKS GPU vLLM LLM GenAI

vLLM pod CrashLoopBackOff with torch.cuda.OutOfMemoryError on EKS: fix VRAM exhaustion

Fix vLLM inference pods crashing with CUDA OutOfMemoryError on EKS. Diagnose KV cache overflow, tune memory parameters, or switch to quantised model weights.

·
vLLM pod entering CrashLoopBackOff with torch.cuda.OutOfMemoryError? The model loaded yesterday but fails today after a config change. The issue is usually KV cache memory exceeding available VRAM - not the model weights themselves.

Symptoms

Pod status shows restart loop:

kubectl get pods -n inference
# NAME                          READY   STATUS             RESTARTS   AGE
# vllm-llama-scout-7b9f4-xyz   0/1     CrashLoopBackOff   5          8m

Logs reveal the OOM:

kubectl logs -n inference vllm-llama-scout-7b9f4-xyz --previous

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has
a total capacity of 23.65 GiB of which 1.12 GiB is free. Including non-PyTorch memory,
this process has 22.53 GiB memory in use.

Or the vLLM-specific variant:

ValueError: The model's max seq len (32768) is larger than the maximum number of tokens
that can be stored in KV cache (8192). Try increasing `gpu_memory_utilization` or
decreasing `max_model_len` when initializing the engine.

Cause

vLLM allocates GPU memory in two parts:

  1. Model weights - fixed size, loaded at startup (e.g. Llama 4 Scout 17B AWQ = ~9GB)
  2. KV cache - dynamic, allocated from remaining VRAM for concurrent request context

The formula: Available VRAM x gpu_memory_utilization - model_weights = KV cache budget

On a g5.xlarge (24GB A10G):

  • 24GB x 0.9 (default utilization) = 21.6GB available
  • Model weights: ~9GB (17B AWQ 4-bit)
  • KV cache: ~12.6GB

OOM happens when:

  • max-model-len too high - setting --max-model-len 32768 pre-allocates KV cache for 32K-token sequences, exhausting VRAM even with few concurrent requests
  • gpu-memory-utilization too high - 0.95 leaves no headroom for PyTorch CUDA allocator overhead
  • Model larger than expected - upgraded from AWQ 4-bit to GPTQ 8-bit without adjusting memory budget
  • Tensor parallelism mismatch - running a model that needs 2 GPUs on a single-GPU node
  • CUDA graphs - vLLM’s CUDA graph compilation reserves additional memory at startup

Fix

Step 1: Check current VRAM usage on the node

If the pod is crashing, check what’s using GPU memory:

# Find the node running the pod
kubectl get pod -n inference vllm-llama-scout-7b9f4-xyz -o wide

# SSH or exec onto node and check nvidia-smi
kubectl debug node/<node-name> -it --image=nvidia/cuda:12.4.0-base-ubuntu22.04 -- nvidia-smi

Expected output shows total/used/free:

| 0  NVIDIA A10G  | 00000000:00:1E.0 | 24576MiB / 24576MiB |

Step 2: Reduce max-model-len (fastest fix)

If you don’t need 32K context, reduce it:

containers:
  - name: vllm
    args:
      - "--model"
      - "meta-llama/Llama-4-Scout-17B-Instruct"
      - "--max-model-len"
      - "4096"            # Reduced from 32768
      - "--gpu-memory-utilization"
      - "0.9"

This dramatically reduces KV cache pre-allocation. For most RAG use cases (prompt + context = 2-4K tokens), 4096 or 8192 is sufficient.

Step 3: Lower gpu-memory-utilization

If you’re at 0.95, drop to 0.85:

      - "--gpu-memory-utilization"
      - "0.85"            # Leave 15% headroom for CUDA overhead

The 10% you “lose” prevents fragmentation-induced OOMs that happen sporadically under load.

Step 4: Disable CUDA graphs (if memory is extremely tight)

CUDA graphs pre-compile execution patterns for speed but consume additional VRAM at startup:

      - "--enforce-eager"   # Disables CUDA graphs, saves ~1-2GB VRAM

Trade-off: ~10-15% slower inference throughput, but the pod will actually start.

Step 5: Switch to more aggressive quantisation

If the model weights themselves are too large:

Quantisation 17B Model Size Quality Impact
FP16 (no quantisation) ~34GB Baseline
GPTQ 8-bit ~17GB Minimal
AWQ 4-bit ~9GB Slight degradation on complex reasoning
GGUF Q4_K_M ~10GB Moderate, good for chat

Switch from FP16 to AWQ:

      - "--model"
      - "meta-llama/Llama-4-Scout-17B-Instruct-AWQ"
      - "--quantization"
      - "awq"

Step 6: Scale to a larger GPU (if the model genuinely needs more VRAM)

If you need full 32K context with a 17B model at FP16, a single A10G (24GB) won’t fit. Options:

  • g5.12xlarge (4x A10G, 96GB total) with --tensor-parallel-size 4
  • g6.xlarge (1x L4, 24GB) - same VRAM but newer GPU, slightly better for inference
  • p4d.24xlarge (8x A100, 320GB) - for 70B+ models

Update your Karpenter NodePool requirements:

requirements:
  - key: "karpenter.k8s.aws/instance-family"
    operator: In
    values: ["g5"]
  - key: "karpenter.k8s.aws/instance-size"
    operator: In
    values: ["12xlarge"]    # Upgraded from xlarge

Validation

After applying the fix, verify the pod starts and VRAM is stable:

# Pod should be Running
kubectl get pods -n inference -l app=vllm-llama-scout
# NAME                          READY   STATUS    RESTARTS   AGE
# vllm-llama-scout-7b9f4-abc   1/1     Running   0          5m

# Check VRAM usage is stable (not climbing)
kubectl exec -n inference vllm-llama-scout-7b9f4-abc -- nvidia-smi --query-gpu=memory.used,memory.total --format=csv
# memory.used [MiB], memory.total [MiB]
# 18432 MiB, 24576 MiB   (75% - healthy)

Send a test request to confirm inference works:

kubectl port-forward -n inference svc/vllm-llama-scout 8000:8000 &
curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-4-Scout-17B-Instruct","prompt":"Hello","max_tokens":10}'

Expected: valid JSON response with generated tokens within 2-3 seconds.