GCP VPC Service Controls security PCI DSS access denied

GCP VPC Service Controls - request blocked by perimeter

Diagnose and fix API requests blocked by GCP VPC Service Controls perimeters: verify access levels, configure ingress/egress rules, analyze VPC-SC audit logs, and maintain PCI DSS compliance.

Jerzy Kopaczewski · 2026-07-31

API calls to BigQuery, Cloud Storage, or KMS fail with PERMISSION_DENIED or Request is prohibited by organisation's policy despite correct IAM permissions. The VPC Service Controls perimeter is actively blocking the request - the calling identity or project is not authorised to cross the perimeter boundary. Services inside the perimeter are unreachable from outside, and any attempt from an unauthorised context generates an organisation-level denial.

This runbook covers diagnosing and fixing situations where VPC Service Controls blocks legitimate API calls. For a detailed overview of GCP security architecture with VPC-SC in PCI DSS compliant environments, see GCP Infrastructure for Fintech - PCI DSS, SOC2 and Google Cloud.

Symptoms

API calls to services protected by a VPC-SC perimeter return authorization errors despite correct IAM permissions:

# BigQuery query returns PERMISSION_DENIED
bq query --project_id=protected-project \
  'SELECT * FROM dataset.table LIMIT 10'
# Error: Access Denied: BigQuery BigQuery: Request is prohibited by
# organization's policy. vpcServiceControlsUniqueIdentifier: abcd1234

# Cloud Storage returns error
gsutil ls gs://protected-bucket/
# AccessDeniedException: 403 Request is prohibited by organization's policy.
# VPC Service Controls: Request is prohibited by organization's policy.
# vpcServiceControlsUniqueIdentifier: efgh5678

# KMS returns denial
gcloud kms keys list \
  --location=europe-west1 \
  --keyring=my-keyring \
  --project=protected-project
# ERROR: (gcloud.kms.keys.list) PERMISSION_DENIED: Request is prohibited
# by organization's policy.

# Terraform plan/apply fails
terraform plan
# Error: Error when reading or editing BigQuery Dataset:
# googleapi: Error 403: Request is prohibited by organization's policy.
# VPC Service Controls: Request is prohibited by organization's policy.

# Service account in Cloud Function cannot read from GCS
# Cloud Function logs:
# google.api_core.exceptions.Forbidden: 403 Request is prohibited by
# organization's policy. vpcServiceControlsUniqueIdentifier: ijkl9012

The key identifier is vpcServiceControlsUniqueIdentifier - it allows you to find the specific denial in Audit Logs and diagnose the root cause.

Causes

VPC Service Controls blocks requests for five main reasons:

Service account or user not in access level. The VPC-SC perimeter requires the caller to satisfy conditions defined in an access level (e.g., specific corporate network, managed device, identity in an allowed list). If the request originates from an IP outside the allowed network or from an identity without an access level - it is blocked.
Project not included in perimeter resources. The project from which the request originates (or the target project) is not added to the perimeter’s resources list. VPC-SC treats such a project as external and blocks communication.
API not in restricted_services list causing implicit deny. If the API service (e.g., bigquery.googleapis.com) is not added to restricted_services of the perimeter, requests to that service from a protected project may be handled differently than expected. The configuration can be unintuitive - some services require explicit inclusion.
Ingress/egress rules misconfigured. Ingress rules control who can enter the perimeter, egress rules control what can leave. Common mistake: missing ingress rule for CI/CD pipeline, Terraform runner, or Cloud Function in a different project.
Using gcloud from outside the perimeter without access level. A developer working from a laptop (outside the corporate network) tries to call an API protected by the perimeter. Without an access level based on identity (not just IP) - the request is blocked.

Solution

Step 1: Find the denial in VPC-SC Audit Logs

# Search VPC-SC denial by unique identifier from the error message
gcloud logging read \
  'protoPayload.metadata.@type="type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata" AND
   protoPayload.metadata.vpcServiceControlsUniqueId="abcd1234"' \
  --project=protected-project \
  --format="json" \
  --limit=5

# Alternatively - find all denials in the last hour
gcloud logging read \
  'protoPayload.metadata.@type="type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata" AND
   protoPayload.metadata.violationReason!=""' \
  --project=protected-project \
  --freshness=1h \
  --format="json(
    timestamp,
    protoPayload.authenticationInfo.principalEmail,
    protoPayload.methodName,
    protoPayload.metadata.violationReason,
    protoPayload.metadata.resourceNames,
    protoPayload.metadata.accessLevels,
    protoPayload.metadata.vpcServiceControlsUniqueId
  )"

# Example output:
# {
#   "timestamp": "2026-07-31T08:15:23Z",
#   "protoPayload": {
#     "authenticationInfo": {"principalEmail": "terraform@ci-project.iam.gserviceaccount.com"},
#     "methodName": "google.storage.objects.list",
#     "metadata": {
#       "violationReason": "NO_MATCHING_ACCESS_LEVEL",
#       "resourceNames": ["projects/_/buckets/protected-bucket"],
#       "accessLevels": []
#     }
#   }
# }

Step 2: Check the perimeter configuration

# List perimeters in the organization
gcloud access-context-manager perimeters list \
  --policy=POLICY_ID \
  --format="table(name, title, status.resources, status.restrictedServices)"

# Get details of the specific perimeter
gcloud access-context-manager perimeters describe my-perimeter \
  --policy=POLICY_ID \
  --format="yaml(name, title, status)"

# Check access levels associated with the perimeter
gcloud access-context-manager levels list \
  --policy=POLICY_ID \
  --format="table(name, title, basic.conditions)"

# Get details of a specific access level
gcloud access-context-manager levels describe my-access-level \
  --policy=POLICY_ID \
  --format="yaml(basic.conditions)"

Step 3: Add identity to access level (for users/service accounts)

# Create or update access level with allowed identities
gcloud access-context-manager levels update my-access-level \
  --policy=POLICY_ID \
  --title="CI/CD and admin access" \
  --basic-level-spec=access-level-spec.yaml

# access-level-spec.yaml file:
cat << 'EOF' > access-level-spec.yaml
- members:
    - serviceAccount:terraform@ci-project.iam.gserviceaccount.com
    - serviceAccount:cloud-function@protected-project.iam.gserviceaccount.com
    - user:admin@company.com
  # Optionally: restrict to specific IPs
  ipSubnetworks:
    - 203.0.113.0/24
    - 198.51.100.0/24
EOF

# Apply the updated access level
gcloud access-context-manager levels update my-access-level \
  --policy=POLICY_ID \
  --basic-level-spec=access-level-spec.yaml

Step 4: Configure ingress rules for external calls

# Ingress rules allow fine-grained control over who can enter the perimeter
# and which services/resources they can access

# Update perimeter with ingress rule
gcloud access-context-manager perimeters update my-perimeter \
  --policy=POLICY_ID \
  --set-ingress-policies=ingress-policy.yaml

# ingress-policy.yaml file:
cat << 'EOF' > ingress-policy.yaml
- ingressFrom:
    identityType: ANY_IDENTITY
    sources:
      - accessLevel: accessPolicies/POLICY_ID/accessLevels/my-access-level
  ingressTo:
    operations:
      - serviceName: storage.googleapis.com
        methodSelectors:
          - method: "*"
      - serviceName: bigquery.googleapis.com
        methodSelectors:
          - method: "*"
    resources:
      - projects/123456789
- ingressFrom:
    identities:
      - serviceAccount:terraform@ci-project.iam.gserviceaccount.com
    sources:
      - resource: projects/987654321
  ingressTo:
    operations:
      - serviceName: storage.googleapis.com
        methodSelectors:
          - method: google.storage.objects.get
          - method: google.storage.objects.list
      - serviceName: cloudkms.googleapis.com
        methodSelectors:
          - method: "*"
    resources:
      - projects/123456789
EOF

Step 5: Add project to perimeter resources (if missing)

# Add a project to the perimeter
gcloud access-context-manager perimeters update my-perimeter \
  --policy=POLICY_ID \
  --add-resources="projects/MISSING_PROJECT_NUMBER"

# Verify the result
gcloud access-context-manager perimeters describe my-perimeter \
  --policy=POLICY_ID \
  --format="yaml(status.resources)"

Step 6: Terraform - ingress rule as code (recommended approach)

# Terraform configuration for VPC Service Controls ingress rule
resource "google_access_context_manager_service_perimeter" "main" {
  parent = "accessPolicies/${var.access_policy_id}"
  name   = "accessPolicies/${var.access_policy_id}/servicePerimeters/main_perimeter"
  title  = "Main Production Perimeter"

  status {
    resources = [
      "projects/${var.protected_project_number}",
    ]

    restricted_services = [
      "bigquery.googleapis.com",
      "storage.googleapis.com",
      "cloudkms.googleapis.com",
    ]

    access_levels = [
      google_access_context_manager_access_level.ci_cd.name,
    ]

    ingress_policies {
      ingress_from {
        identity_type = "ANY_IDENTITY"
        sources {
          access_level = google_access_context_manager_access_level.ci_cd.name
        }
      }
      ingress_to {
        resources = ["projects/${var.protected_project_number}"]
        operations {
          service_name = "bigquery.googleapis.com"
          method_selectors {
            method = "*"
          }
        }
        operations {
          service_name = "storage.googleapis.com"
          method_selectors {
            method = "*"
          }
        }
      }
    }

    ingress_policies {
      ingress_from {
        sources {
          resource = "projects/${var.ci_cd_project_number}"
        }
        identities = [
          "serviceAccount:terraform@${var.ci_cd_project_id}.iam.gserviceaccount.com",
        ]
      }
      ingress_to {
        resources = ["projects/${var.protected_project_number}"]
        operations {
          service_name = "storage.googleapis.com"
          method_selectors {
            method = "google.storage.objects.get"
          }
          method_selectors {
            method = "google.storage.objects.list"
          }
          method_selectors {
            method = "google.storage.objects.create"
          }
        }
        operations {
          service_name = "cloudkms.googleapis.com"
          method_selectors {
            method = "*"
          }
        }
      }
    }
  }
}

resource "google_access_context_manager_access_level" "ci_cd" {
  parent = "accessPolicies/${var.access_policy_id}"
  name   = "accessPolicies/${var.access_policy_id}/accessLevels/ci_cd_access"
  title  = "CI/CD Pipeline Access"

  basic {
    conditions {
      members = [
        "serviceAccount:terraform@${var.ci_cd_project_id}.iam.gserviceaccount.com",
        "serviceAccount:deployer@${var.ci_cd_project_id}.iam.gserviceaccount.com",
      ]
      ip_subnetworks = var.allowed_cidr_ranges
    }
  }
}

Step 7: Use dry-run mode to test changes (without blocking production)

# Use dry-run mode to test new perimeter configuration
# Dry-run logs violations but does not block requests
gcloud access-context-manager perimeters dry-run create my-perimeter-test \
  --policy=POLICY_ID \
  --resources="projects/123456789" \
  --restricted-services="bigquery.googleapis.com,storage.googleapis.com" \
  --ingress-policies=ingress-policy.yaml

# Check dry-run violations in logs
gcloud logging read \
  'protoPayload.metadata.@type="type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata" AND
   protoPayload.metadata.dryRun=true' \
  --project=protected-project \
  --freshness=1h \
  --format="json(timestamp, protoPayload.metadata.violationReason)"

# After confirming no false positives - enforce the configuration
gcloud access-context-manager perimeters dry-run enforce my-perimeter \
  --policy=POLICY_ID

Validation

After applying perimeter configuration changes, verify that legitimate requests pass through:

# 1. Retry the request that was previously blocked
gsutil ls gs://protected-bucket/
# Expected: file listing (no 403 error)

bq query --project_id=protected-project \
  'SELECT 1 AS test'
# Expected: query result returned

# 2. Check for new VPC-SC denials in logs
gcloud logging read \
  'protoPayload.metadata.@type="type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata" AND
   protoPayload.metadata.violationReason!="" AND
   protoPayload.authenticationInfo.principalEmail="terraform@ci-project.iam.gserviceaccount.com"' \
  --project=protected-project \
  --freshness=30m
# Expected: no results

# 3. Verify perimeter still protects against unauthorized access
# (negative test - request from unrelated service account should be blocked)
gcloud auth activate-service-account unauthorized@other-project.iam.gserviceaccount.com \
  --key-file=unauthorized-key.json
gsutil ls gs://protected-bucket/
# Expected: AccessDeniedException 403 (perimeter still protects)

# 4. Check full perimeter status
gcloud access-context-manager perimeters describe my-perimeter \
  --policy=POLICY_ID \
  --format="yaml(status)"
# Confirm resources, restrictedServices, ingressPolicies are correct

# 5. Run Terraform plan - confirm no drift (idempotent)
terraform plan
# Expected: No changes. Your infrastructure matches the configuration.

VPC Service Controls is the foundation of data security in GCP for regulated environments (PCI DSS, SOC2, HIPAA). A denial means the protection is working - the problem is in the access configuration, not in the security mechanism itself. Never remove a perimeter or broaden it excessively to fix a single denial. Instead, create precise ingress rules with minimal scope (least privilege). Every perimeter change should go through dry-run mode before being applied to production, to avoid blocking legitimate traffic.

VPC Service Controls blocking your GCP APIs?

Book a free 30-minute call. We design perimeter configurations with precise ingress rules, access levels, and PCI DSS compliance without blocking your pipeline.