Logo
GCP GKE Kubernetes autoscaling

GKE node pool stuck in PROVISIONING: quota exhaustion, IP ranges, and machine type availability

Resolve GKE node pool stuck in PROVISIONING state caused by quota exhaustion, insufficient pod IP ranges, or machine type unavailability in the target zone.

·
Your GKE node pool is stuck in PROVISIONING or new nodes are NotReady. Pods are pending, autoscaler is trying to scale up, but no new nodes appear. The cause is almost always a resource constraint outside of Kubernetes itself - GCP quotas, IP address exhaustion, or machine type availability.

Symptoms

Node pool shows status PROVISIONING for extended periods. Pods remain in Pending state with scheduling failures:

# Node pool stuck provisioning
gcloud container node-pools describe default-pool \
  --cluster my-cluster \
  --zone europe-west1-b \
  --format="value(status)"
# Output: PROVISIONING

# Nodes in NotReady state
kubectl get nodes
# NAME                                  STATUS     AGE   VERSION
# gke-my-cluster-default-pool-abc123    NotReady   15m   v1.29.4-gke.1234

# Pods stuck Pending
kubectl get pods -A | grep Pending
# default   my-app-7b9f4d-xk2lm   0/1   Pending   0   12m

# Check why pods are not scheduled
kubectl describe pod my-app-7b9f4d-xk2lm | grep -A5 Events
# Events:
#   Warning  FailedScheduling  pod/my-app-7b9f4d-xk2lm
#   0/3 nodes are available: 3 Insufficient cpu.

# Cluster autoscaler logs
kubectl -n kube-system logs -l k8s-app=cluster-autoscaler --tail=50 | grep -i "scale up"
# Could not scale up node pool: RESOURCE_POOL_EXHAUSTED

The cluster autoscaler wants to add nodes but cannot create them due to infrastructure-level constraints.

Cause

GKE node provisioning fails for three main infrastructure reasons:

  • Regional quota exhaustion. GCP enforces per-region quotas on CPUs, instances, IP addresses, and GPU accelerators. When you hit the limit, Compute Engine refuses to create new VMs. This is the most common cause in projects that run multiple clusters or have grown organically.
  • Insufficient pod IP range. GKE uses secondary IP ranges for pods. Each node reserves a /24 block (256 IPs) by default. If the secondary range is too small, GKE cannot assign a pod CIDR to the new node and provisioning stalls.
  • Machine type unavailable in the zone. Not all machine types are available in all zones. If you request n2-standard-32 in a zone where that family is sold out, provisioning waits indefinitely. This is common with GPU instances and large machine types.

Fix

Step 1: Check quota exhaustion

# List all quotas for the region with current usage
gcloud compute regions describe europe-west1 \
  --format="table(quotas.metric,quotas.limit,quotas.usage)" \
  | grep -E "CPUS|INSTANCES|IN_USE_ADDRESSES|SSD_TOTAL_GB"

# Example output showing CPUS quota exhausted:
# CPUS                24.0    24.0    <-- at limit
# INSTANCES           100.0   15.0
# IN_USE_ADDRESSES    23.0    20.0
# SSD_TOTAL_GB        2048.0  1500.0

# Check if the specific quota is blocking
gcloud compute regions describe europe-west1 \
  --format="json(quotas)" | jq '.quotas[] | select(.usage >= .limit)'

If quotas are exhausted, request an increase:

# Request quota increase via gcloud
gcloud compute regions describe europe-west1 --format="json(quotas)"

# Then request increase in Console:
# IAM & Admin > Quotas > Filter by region/metric > Edit Quotas
# Or use gcloud:
gcloud alpha services quota update \
  --consumer=projects/my-project \
  --service=compute.googleapis.com \
  --metric=compute.googleapis.com/cpus \
  --unit=1/{project}/{region} \
  --dimensions=region=europe-west1 \
  --value=48

Step 2: Check pod IP range exhaustion

# Check secondary ranges on the subnet
gcloud compute networks subnets describe my-subnet \
  --region=europe-west1 \
  --format="table(secondaryIpRanges.rangeName,secondaryIpRanges.ipCidrRange)"

# Example output:
# RANGE_NAME     IP_CIDR_RANGE
# pods           10.4.0.0/20      <-- Only 16 /24 blocks = max 16 nodes
# services       10.8.0.0/20

# Check how many nodes are consuming pod CIDRs
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'

If the pod range is full, expand it or create a new node pool with a larger range:

# Option 1: Create a new node pool with a different subnet that has larger pod range
gcloud container node-pools create expanded-pool \
  --cluster=my-cluster \
  --zone=europe-west1-b \
  --machine-type=e2-standard-4 \
  --num-nodes=3 \
  --max-pods-per-node=64 \
  --enable-autoscaling \
  --min-nodes=1 \
  --max-nodes=10

# Option 2: Reduce max-pods-per-node to use smaller CIDR blocks per node
# This allows more nodes within the same secondary range
gcloud container node-pools create efficient-pool \
  --cluster=my-cluster \
  --zone=europe-west1-b \
  --machine-type=e2-standard-4 \
  --max-pods-per-node=32 \
  --num-nodes=3

Step 3: Handle machine type unavailability

# Check which machine types are available in the zone
gcloud compute machine-types list \
  --zones=europe-west1-b \
  --filter="name:n2-standard" \
  --format="table(name,zone,guestCpus,memoryMb)"

# If the machine type is unavailable, check other zones
gcloud compute machine-types list \
  --filter="name=n2-standard-8" \
  --format="table(name,zone)"

Use multi-zone node pools with fallback machine types:

# Create a regional (multi-zone) node pool with machine type fallback
gcloud container node-pools create resilient-pool \
  --cluster=my-cluster \
  --region=europe-west1 \
  --machine-type=n2-standard-8 \
  --enable-autoscaling \
  --min-nodes=1 \
  --max-nodes=10 \
  --num-nodes=3

# For even more resilience, use compact placement with spot VMs
gcloud container node-pools create spot-pool \
  --cluster=my-cluster \
  --region=europe-west1 \
  --machine-type=n2-standard-8 \
  --spot \
  --enable-autoscaling \
  --min-nodes=0 \
  --max-nodes=20 \
  --node-labels=workload-type=batch

Step 4: Configure node auto-provisioning (NAP) for automatic fallback

# Enable node auto-provisioning with resource limits
gcloud container clusters update my-cluster \
  --zone=europe-west1-b \
  --enable-autoprovisioning \
  --min-cpu=1 \
  --max-cpu=100 \
  --min-memory=1 \
  --max-memory=400 \
  --autoprovisioning-locations=europe-west1-b,europe-west1-c,europe-west1-d

NAP lets GKE automatically create node pools with different machine types when the preferred type is unavailable.

Validation

After applying fixes, confirm nodes can be provisioned:

# 1. Verify node pool is RUNNING (not PROVISIONING)
gcloud container node-pools describe default-pool \
  --cluster my-cluster \
  --zone europe-west1-b \
  --format="value(status)"
# Expected: RUNNING

# 2. Verify all nodes are Ready
kubectl get nodes
# Expected: all STATUS = Ready

# 3. Verify pending pods are now scheduled
kubectl get pods -A | grep Pending
# Expected: no results

# 4. Verify autoscaler can create new nodes (trigger a scale-up)
kubectl run test-scale --image=nginx --requests='cpu=500m' --replicas=5
kubectl get pods -w
# Expected: pods schedule within 2-3 minutes as new nodes come up

# Clean up test
kubectl delete deployment test-scale

# 5. Check cluster autoscaler status
kubectl -n kube-system describe configmap cluster-autoscaler-status | grep -A5 "Health"
# Expected: Healthy

A GKE cluster that cannot scale is a ticking time bomb. When traffic spikes or deploys demand more resources, pods stay Pending and your application becomes unavailable. Quota increases take minutes to request but hours to approve - so plan ahead. Keep at least 20% headroom on CPU and IP quotas for your production regions, and always use multi-zone node pools to survive single-zone capacity issues.

 

Jerzy Kopaczewski

Running GKE in production?

Book a free 30-minute call. We design GKE clusters with proper autoscaling, IP planning, and multi-zone resilience so you never hit provisioning walls.