Logo
GCP Vertex AI machine learning MLOps

Vertex AI endpoint deployment stuck or failed: model container, readiness, and quota

Resolve a Vertex AI DeployModel operation stuck in progress or failing: diagnose the model server container contract, health and predict route readiness, model artifact paths, and the custom-model-serving quota that silently blocks deployment.

Jerzy Kopaczewski ·
Your DeployModel operation on a Vertex AI endpoint runs for a long time and then fails, or never reaches a serving state. The model uploads fine but will not come up behind the endpoint. The cause is almost always one of three: the serving container does not meet Vertex AI's contract, the model artifact or health check is wrong, or the deployment is blocked by a custom-model-serving quota. This runbook separates the three and fixes each.

This runbook covers custom model deployment failures. For how Vertex AI endpoints are billed while deployed - which is why a stuck or forgotten deployment costs money - see how Vertex AI pricing works and the platform overview in what Vertex AI is. For hands-on help, book a consulting session.

Symptoms

The deployment either hangs or returns a failed long-running operation. Identify your shape first.

# 1. DeployModel operation fails after a long wait
google.api_core.exceptions.FailedPrecondition: 400 Model server
never became ready. Please validate that your model file or
container configuration are valid.

# 2. Operation blocked by quota (see the dedicated quota runbook)
google.api_core.exceptions.ResourceExhausted: 429 The following
quotas are exceeded: CustomModelServingT4GPUsPerProjectPerRegion

# 3. Operation just sits in progress
# gcloud ai operations describe shows done: false for 20-30+ minutes
# with no error, then eventually times out.

Observable impact:

  • The model uploads to the Model Registry but never serves traffic
  • gcloud ai endpoints describe shows the endpoint with no deployed model, or a deployed model stuck at 0 healthy replicas
  • The DeployModel long-running operation eventually fails or times out

Cause

A Vertex AI custom model has to pass a container contract and a readiness check before the endpoint will route to it, and the whole operation runs inside a quota envelope.

  • Container contract not met. A custom serving container must listen on the port Vertex AI expects, answer the configured health route, and answer the predict route. If any of these is wrong, the model server “never becomes ready” and the deploy fails after the timeout.
  • Model artifact or route mismatch. For prebuilt containers, the artifact at the artifactUri must match the framework’s expected layout (for example a model.joblib, saved_model.pb, or the files the container expects). A wrong path or format means the server starts but cannot load the model.
  • Quota blocks the deployment. Custom model serving consumes a per-region, per-accelerator quota. If the deployment needs a GPU the project has no quota for, or the sum of replicas across deployments exceeds the quota, DeployModel fails with ResourceExhausted. That specific failure mode has its own runbook (linked below).

Fix

Step 1: Read the operation and the model-server logs

The long-running operation carries the failure reason, and Cloud Logging has the container’s own output.

# List recent endpoint operations and their state
gcloud ai operations list --region=us-central1 \
  --filter="done=false" 2>/dev/null

# Describe the specific DeployModel operation (id from the deploy call)
gcloud ai operations describe OPERATION_ID --region=us-central1

# Read the model server container logs in Cloud Logging
gcloud logging read \
  'resource.type="aiplatform.googleapis.com/Endpoint" severity>=WARNING' \
  --limit=50 --freshness=1h --format="value(textPayload)"

If the logs say the model server never became ready, go to Step 2. If you see ResourceExhausted / a CustomModelServing... quota, go to the quota runbook. If the container logs a load error, go to Step 3.

Step 2: Validate the serving container contract locally

Do not debug the contract by redeploying - it is slow and burns quota. Run the container locally and exercise the health and predict routes first.

# Run the serving container locally, mapping the port Vertex AI will probe
docker run --rm -p 8080:8080 \
  -e AIP_HTTP_PORT=8080 \
  -e AIP_HEALTH_ROUTE=/health \
  -e AIP_PREDICT_ROUTE=/predict \
  YOUR_SERVING_IMAGE:TAG

# In another shell: the health route must return 200
curl -i http://localhost:8080/health
# Expected: HTTP/1.1 200

# The predict route must accept the Vertex AI request envelope
curl -s -X POST http://localhost:8080/predict \
  -H "Content-Type: application/json" \
  -d '{"instances":[{"x":1}]}'
# Expected: {"predictions":[...]}

Vertex AI injects AIP_HTTP_PORT, AIP_HEALTH_ROUTE, and AIP_PREDICT_ROUTE into the container; your server must read those env vars rather than hard-coding a port or path. A container that passes these three checks locally will almost always deploy.

Step 3: Confirm the model artifact matches the container

For a prebuilt container, the artifact layout is fixed. Confirm the file the container expects is actually at the artifactUri.

# Inspect what you uploaded
gcloud ai models describe MODEL_ID --region=us-central1 \
  --format="value(containerSpec.imageUri, artifactUri)"

# List the artifact directory in GCS
gcloud storage ls -r gs://your-bucket/your-model-dir/
# Prebuilt sklearn expects: model.joblib or model.pkl
# Prebuilt TensorFlow expects: a SavedModel (saved_model.pb + variables/)
# Prebuilt XGBoost expects: model.bst / model.joblib / model.pkl

If the layout is wrong, re-upload the artifact in the expected format and re-run the deploy.

Step 4: Redeploy with a small, explicit configuration

Once the container passes locally and the artifact is correct, redeploy with a single small replica so a failure surfaces fast and cheap.

gcloud ai endpoints deploy-model ENDPOINT_ID \
  --region=us-central1 \
  --model=MODEL_ID \
  --display-name=my-model-v1 \
  --machine-type=n1-standard-2 \
  --min-replica-count=1 \
  --max-replica-count=1 \
  --traffic-split=0=100

Deploy to a single replica first, confirm it serves, then raise --max-replica-count. Raising replicas before the model serves at all just multiplies the cost of a broken deployment.

Validation

Confirm the model is deployed and serving.

# 1. The endpoint now lists a deployed model with healthy replicas
gcloud ai endpoints describe ENDPOINT_ID --region=us-central1 \
  --format="value(deployedModels[].id, deployedModels[].displayName)"
# Expected: your model id and display name

# 2. A real prediction returns a result
gcloud ai endpoints predict ENDPOINT_ID --region=us-central1 \
  --json-request=request.json
# Expected: a predictions payload, no error

A Vertex AI endpoint bills for every hour a model stays deployed, even with zero traffic. A deployment that is “stuck” is often a model that failed to come up but still reserved capacity - so fix it or undeploy it promptly rather than leaving it to accrue cost.

 

Jerzy Kopaczewski

Running models on Vertex AI?

Book a free 30-minute call. We help teams get custom models serving on Vertex AI, pass the container contract first time, and keep endpoint cost under control.

Book a call