Vertex AI deployment quota exceeded or GPU unavailable: CustomModelServing quotas by region

This runbook covers deploy-time accelerator quota and availability. If your model is already deployed and live traffic is being throttled, that is the prediction 429 runbook instead. For GPU cost planning, see how Vertex AI pricing works. For hands-on help, book a consulting session.

Symptoms

The DeployModel call fails with a quota or availability error. The metric name is the key clue.

# Accelerator quota exceeded for this project and region
google.api_core.exceptions.ResourceExhausted: 429 The following
quotas are exceeded: CustomModelServingT4GPUsPerProjectPerRegion

# Zero quota granted for the accelerator (common on new projects / free tier)
The following quotas are exceeded:
CustomModelServingL4GPUsPerProjectPerRegion  (limit: 0)

# Accelerator simply not available in the chosen region
# Deploy stalls or fails with a capacity/availability error for the
# requested machine-type + accelerator combination in that region.

Observable impact:

• A deploy that worked in one region fails in another with the same config
• A GPU deployment fails while a CPU-only deployment of the same model succeeds
• A brand-new project cannot deploy any GPU model because the accelerator quota is 0

Cause

Vertex AI custom model serving draws on a per-project, per-region, per-accelerator quota, and not every accelerator exists in every region.

• Accelerator quota is per region and per accelerator type. The quota metric looks like CustomModelServing<ACCELERATOR>PerProjectPerRegion - for example CustomModelServingT4GPUsPerProjectPerRegion. Each GPU type (T4, L4, A100, and so on) has its own counter in each region. Hitting it blocks the deploy regardless of your overall GPU quota elsewhere.
• New projects often start at zero. On new projects, and under some free-tier conditions, the serving quota for a given accelerator can be 0, so the first GPU deploy fails until you request an increase.
• The accelerator is not offered in the region. Not every accelerator is available in every region. Requesting a type the region does not offer fails or stalls no matter how much quota you hold.

Fix

Step 1: Read the exact quota from the error

The metric name in the error tells you precisely which counter is blocking you. Note the accelerator and the region.

CustomModelServingT4GPUsPerProjectPerRegion
                   ^^              ^^^^^^^^^^
                   accelerator     scope: per project, per region

Step 2: Check the current quota and usage

Look up the specific metric in the Quotas console or via gcloud, scoped to your region.

# Console path: IAM & Admin > Quotas
#   Service = "Vertex AI API"
#   Filter the metric name from the error, e.g. CustomModelServingT4GPUsPerProjectPerRegion
#   Dimension region = us-central1

# Or list serving quotas via gcloud (names contain CustomModelServing)
gcloud alpha services quota list \
  --service=aiplatform.googleapis.com \
  --consumer=projects/YOUR_PROJECT 2>/dev/null | grep -i customModelServing

# Cross-check Compute Engine GPU quota in the region as well
gcloud compute regions describe us-central1 \
  --format="table(quotas.metric,quotas.limit,quotas.usage)" | grep -i gpu

If the limit is 0 or at usage, go to Step 3 (raise) or Step 4 (relocate).

Step 3: Request a quota increase

If the region and accelerator are right but the quota is too low or 0, request an increase for that exact metric.

# Console: IAM & Admin > Quotas > select the CustomModelServing<ACCEL>...
# metric for your region > Edit Quotas > submit the requested value.
# Increases are reviewed by Google and can take time - plan ahead.

Request only the accelerator and region you actually deploy to. A raise on T4...us-central1 does nothing for an L4...europe-west4 deployment.

Step 4: Switch accelerator, region, or use a reservation

If the quota cannot be raised in time, or the accelerator is unavailable in the region, change the target rather than waiting.

# Option A: deploy to a region that offers the accelerator and has quota
gcloud ai endpoints deploy-model ENDPOINT_ID \
  --region=europe-west4 \
  --model=MODEL_ID \
  --machine-type=g2-standard-8 \
  --accelerator=type=nvidia-l4,count=1 \
  --min-replica-count=1 --max-replica-count=1

# Option B: use a different accelerator the region does have quota for
#   (for example L4 instead of T4, or A100 for larger models)

# Option C: for scarce capacity, deploy with Spot VMs or a reservation
#   so you are not blocked waiting on on-demand availability

For a model that does not strictly need a GPU, a CPU machine type side-steps the accelerator quota entirely - worth checking before chasing a GPU increase.

Validation

Confirm the deploy now completes on the chosen accelerator and region.

# The deploy operation completes without ResourceExhausted
gcloud ai endpoints deploy-model ENDPOINT_ID \
  --region=us-central1 --model=MODEL_ID \
  --machine-type=n1-standard-4 \
  --accelerator=type=nvidia-tesla-t4,count=1 \
  --min-replica-count=1 --max-replica-count=1
# Expected: operation finishes, no CustomModelServing quota error

# The endpoint shows the deployed model
gcloud ai endpoints describe ENDPOINT_ID --region=us-central1 \
  --format="value(deployedModels[].id)"
# Expected: a deployed model id

GPU serving capacity is the most expensive line in a Vertex AI deployment and the scarcest. Request quota only for the accelerators and regions you will actually use, and prefer the smallest accelerator that meets your latency target - an oversized GPU quota is both harder to get approved and more expensive to run.

Planning GPU capacity on Vertex AI?

Book a free 30-minute call. We help teams pick the right accelerator and region, plan quota ahead of launch, and decide between on-demand, Spot, and reservations.

Book a call Send us a message

Related

• Vertex AI endpoint deployment stuck or failed - other deploy failure modes (container, artifact, readiness)
• Vertex AI prediction 429 / RESOURCE_EXHAUSTED - the runtime version of a quota ceiling
• How Vertex AI pricing works - GPU serving as the costliest line item
• Vertex AI quotas and limits documentation