Vertex AI online prediction 429 RESOURCE_EXHAUSTED: quota, autoscaling, and retries

This runbook covers prediction-time throttling. If the model never deployed in the first place, see the endpoint deployment runbook. For how prediction capacity maps to cost, see how Vertex AI pricing works. For hands-on help, book a consulting session.

Symptoms

The call returns a 429. The message text points to which cause you have.

# Generic resource exhaustion (often transient, shared capacity)
google.api_core.exceptions.ResourceExhausted: 429 Resource exhausted.
Please try again later.

# Replica / serving-quota ceiling (structural, not transient)
google.api_core.exceptions.ResourceExhausted: 429 The following
quotas are exceeded: CustomModelServing<ACCELERATOR>PerProjectPerRegion

Observable impact:

• A share of prediction calls fail under load while others succeed
• Failures cluster at traffic peaks and disappear when traffic drops (transient) - or persist regardless of time (quota ceiling)
• Async or high-concurrency clients see 429s far more often than a simple sequential client

Cause

Two mechanisms produce a 429 on online prediction, and the fix depends on which one you are hitting.

• Shared on-demand capacity throttling. On-demand prediction competes for shared GPU/TPU capacity in the region. When demand in that region spikes, on-demand requests are the first to be throttled. These 429s are transient and are meant to be retried. They are more common with asynchronous, high-concurrency clients and large multimodal requests.
• Autoscaling replica-quota ceiling. Quota for custom model serving is based on the real-time compute your deployed models use. If the sum of maxReplicaCount across all deployments in the project exceeds the project’s quota, deployments cannot autoscale up to meet load, so requests beyond current capacity are rejected. This is structural - no amount of retrying clears it until the quota or the replica ceiling changes.

Fix

Step 1: Decide which 429 you have

Check whether the endpoint is hitting its replica ceiling or just riding shared capacity.

# How many replicas is the deployed model allowed and using?
gcloud ai endpoints describe ENDPOINT_ID --region=us-central1 \
  --format="yaml(deployedModels[].dedicatedResources.minReplicaCount,
    deployedModels[].dedicatedResources.maxReplicaCount)"

# Look at replica count and rejection metrics in Cloud Monitoring
# (prediction/online/replica_count and prediction error counts)
gcloud monitoring time-series list \
  --filter='metric.type="aiplatform.googleapis.com/prediction/online/replica_count"' \
  --interval-start-time="$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" 2>/dev/null

If replica count is pinned at maxReplicaCount during failures, you have a capacity/quota ceiling (Step 2). If replicas are below max but you still get 429s, it is shared-capacity throttling (Step 3).

Step 2: Raise the replica ceiling or the serving quota

If the model is pinned at max replicas, give it room - but only up to the project’s serving quota. If the quota itself is the limit, request an increase.

# Raise the deployed model's max replica count
gcloud ai endpoints deploy-model ENDPOINT_ID \
  --region=us-central1 \
  --model=MODEL_ID \
  --max-replica-count=10 \
  --traffic-split=0=100

# Check the custom-model-serving quota before raising replicas past it
# Console: IAM & Admin > Quotas, filter Service = "Vertex AI API",
# metric CustomModelServing<ACCELERATOR>PerProjectPerRegion
gcloud alpha services quota list \
  --service=aiplatform.googleapis.com \
  --consumer=projects/YOUR_PROJECT 2>/dev/null | grep -i customModelServing

Raising maxReplicaCount above the serving quota does not help - the deployment still cannot scale past the quota. Request a quota increase from the Quotas page when the quota is the true ceiling.

Step 3: Add exponential backoff for transient 429s

Shared-capacity 429s are expected under load and should be retried with backoff, not surfaced to users.

import time, random
from google.api_core.exceptions import ResourceExhausted

def predict_with_retry(client, endpoint, instances, max_retries=5):
    for attempt in range(max_retries):
        try:
            return client.predict(endpoint=endpoint, instances=instances)
        except ResourceExhausted:
            if attempt == max_retries - 1:
                raise
            # exponential backoff with jitter
            time.sleep(min(2 ** attempt, 30) + random.uniform(0, 1))

Keep concurrency sane. Fan-out async clients amplify 429s; cap in-flight requests and spread load rather than firing everything at once.

Step 4: For steady, predictable load, reserve capacity

If the workload is continuous rather than bursty, move off pure on-demand so you are not competing for shared capacity.

• Use a dedicated endpoint with a higher minReplicaCount so warm capacity is always held
• For generative models, consider Provisioned Throughput where available, which reserves capacity instead of sharing it
• Keep the endpoint and its callers in the same region to avoid cross-region contention

Validation

Confirm the 429 rate drops under representative load.

# Drive a short, representative burst and count non-200 responses
for i in $(seq 1 50); do
  gcloud ai endpoints predict ENDPOINT_ID --region=us-central1 \
    --json-request=request.json >/dev/null 2>err.txt || cat err.txt
done | grep -c "RESOURCE_EXHAUSTED"
# Expected: 0 (or a small number fully absorbed by backoff retries)

Then confirm replicas now scale under load rather than pinning at the old ceiling:

gcloud ai endpoints describe ENDPOINT_ID --region=us-central1 \
  --format="value(deployedModels[].dedicatedResources.maxReplicaCount)"
# Expected: the raised ceiling, with replicas scaling below it

Every replica you hold warm to beat 429s also bills by the hour whether it serves traffic or not. Size minReplicaCount to real baseline load, not to peak - over-provisioning to avoid occasional throttling is often more expensive than a correct backoff.

Scaling prediction traffic on Vertex AI?

Book a free 30-minute call. We help teams size Vertex AI endpoints for real load, tune autoscaling, and decide when reserved capacity beats on-demand.

Book a call Send us a message

Related

• Vertex AI endpoint deployment stuck or failed - when the model never came up in the first place
• Vertex AI deployment quota exceeded / GPU unavailable - the deploy-time version of a quota ceiling
• How Vertex AI pricing works - why warm replicas cost money even when idle
• Vertex AI prediction autoscaling documentation