Vertex AI prediction 403 permission denied: aiplatform.endpoints.predict and the service account

This runbook covers the prediction permission error. If the endpoint itself will not deploy, see the deployment runbook; if calls are rejected with 429 rather than 403, see the prediction 429 runbook. For hands-on help, book a consulting session.

Symptoms

The call returns a 403 naming the aiplatform.endpoints.predict permission and the resource it was denied on.

google.api_core.exceptions.PermissionDenied: 403 Permission
'aiplatform.endpoints.predict' denied on resource
'//aiplatform.googleapis.com/projects/PROJECT/locations/us-central1/endpoints/ENDPOINT_ID'
(or it may not exist).

Observable impact:

• The same request works from your own user account (which has broad roles) but fails from a service account or CI job
• The endpoint is visible and healthy in the console, yet the API rejects the call
• The “(or it may not exist)” tail hints the identity may also be pointed at the wrong project or path

Cause

The predict call is an IAM-checked operation, and a 403 means the authenticated identity fails that check on that resource.

• The calling identity lacks the role. Calling predict requires the aiplatform.endpoints.predict permission, carried by the Vertex AI User role (roles/aiplatform.user). A service account created for the app, or a CI/CD identity, usually does not have it by default.
• Wrong project or credentials. If the request runs with Application Default Credentials for a different project, or a key for the wrong service account, the identity may simply not exist on the target project - which is exactly what the “(or it may not exist)” wording covers.
• Wrong resource path. A mistyped project, location, or endpoint id denies on a resource the identity has no grant on, producing the same 403 even when the IAM binding is otherwise correct.

Fix

Step 1: Identify the calling identity

Find out exactly which identity the failing request authenticates as - fix the wrong one and you will chase this for hours.

# Which identity are the ambient credentials?
gcloud auth list --format="value(account)"

# If the app runs as a service account, confirm which one
# (e.g. on GKE via Workload Identity, or an attached VM/Run SA)
gcloud run services describe YOUR_SERVICE --region=us-central1 \
  --format="value(spec.template.spec.serviceAccountName)" 2>/dev/null

Note the email of the identity actually making the call - that is the principal that needs the grant.

Step 2: Grant the Vertex AI User role

Bind roles/aiplatform.user to that identity on the project that owns the endpoint.

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:your-sa@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"

Prefer this predefined role over hand-crafting a custom role; it carries aiplatform.endpoints.predict plus the related read permissions a caller normally needs. IAM changes can take a minute or two to propagate.

Step 3: Confirm the project and resource path

Rule out the “wrong project / wrong path” cause before assuming IAM is still broken.

# Which project do the ambient credentials resolve to?
gcloud config get-value project

# Does the endpoint actually exist at the path in the error?
gcloud ai endpoints describe ENDPOINT_ID --region=us-central1 \
  --project=PROJECT_ID --format="value(name)"
# Expected: the full endpoint resource name - if this 404s, your
# path or project is wrong, not your IAM.

Make sure the project, location, and endpoint id in your client exactly match this resource name.

Step 4: Verify the grant resolves the permission

Test that the identity now has the permission, without running a full prediction.

# Impersonate the service account and test the permission
gcloud ai endpoints describe ENDPOINT_ID --region=us-central1 \
  --project=PROJECT_ID \
  --impersonate-service-account=your-sa@PROJECT_ID.iam.gserviceaccount.com
# Expected: the endpoint describes successfully under that identity

Validation

Confirm a real prediction now succeeds as the intended identity.

# Run a prediction authenticated as the service account
gcloud ai endpoints predict ENDPOINT_ID --region=us-central1 \
  --project=PROJECT_ID \
  --json-request=request.json \
  --impersonate-service-account=your-sa@PROJECT_ID.iam.gserviceaccount.com
# Expected: a predictions payload, no 403

# Confirm the binding is in place
gcloud projects get-iam-policy PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.role=roles/aiplatform.user" \
  --format="value(bindings.members)" | grep your-sa
# Expected: the service account appears

Grant roles/aiplatform.user to the specific service account that calls the endpoint, not to broad groups or user accounts as a shortcut. Over-broad grants on Vertex AI resources are a common audit finding - least privilege here is both safer and simpler to reason about.

Securing Vertex AI access?

Book a free 30-minute call. We help teams design least-privilege IAM for Vertex AI, wire service accounts correctly with Workload Identity, and keep access audit-ready.

Book a call Send us a message

Related

• Vertex AI endpoint deployment stuck or failed - when the endpoint itself will not come up
• Vertex AI prediction 429 / RESOURCE_EXHAUSTED - rejections that are throttling, not permissions
• What is Vertex AI? - the platform and how endpoints are accessed
• Vertex AI IAM permissions documentation