GKE Binary Authorization - deployment denied by missing attestation
Resolve GKE Binary Authorization deployment denials caused by missing attestations, wrong attestor key version, image digest mismatch, or missing break-glass annotation for emergency deploys.
denied by attestor or Denied by default admission rule. No attestations found. The CI/CD pipeline is blocked, kubectl apply returns an admission error. The image exists in the registry and passes all checks - but Binary Authorization refuses to admit the pod because no valid attestation confirms the image passed required security validations.
This runbook covers diagnosis and resolution of GKE Binary Authorization blocking deployments due to missing or invalid image attestations. For a detailed overview of GCP architecture with Binary Authorization securing the supply chain in PCI DSS compliant environments, see GCP Infrastructure for Fintech - PCI DSS, SOC2 and Google Cloud.
Symptoms
Deployments to a GKE cluster with Binary Authorization policy enabled are rejected. The error appears in kubectl apply output or CI/CD pipeline logs:
# kubectl apply returns admission webhook error
kubectl apply -f deployment.yaml
# Error from server (VIOLATES_POLICY): error when creating "deployment.yaml":
# admission webhook "imagepolicywebhook.image-policy.k8s.io" denied the request:
# Image gcr.io/my-project/my-app@sha256:abc123... denied by attestor
# projects/my-project/attestors/build-attestor: No attestations found
# Alternative message with default deny rule
kubectl apply -f deployment.yaml
# Error from server: error when creating "deployment.yaml":
# admission webhook "imagepolicywebhook.image-policy.k8s.io" denied the request:
# Denied by default admission rule. No attestations found that were valid
# and signed by a key trusted by the attestor
# Check recent events
kubectl get events --sort-by='.lastTimestamp' | grep -i denied
# LAST SEEN TYPE REASON OBJECT MESSAGE
# 2m Warning Denied pod/my-app-7b9f4d-xk2lm Denied by BinAuthz
# Admission controller logs
kubectl logs -n kube-system -l app=binauthz-admission-controller --tail=20
# "msg":"image policy webhook denied","image":"gcr.io/my-project/my-app:v1.2.3",
# "reason":"no valid attestation found for image"
This only affects clusters with an active Binary Authorization policy. The image is correctly built and available in the registry - the block happens at the admission control layer.
Causes
Binary Authorization rejects deployments for four main reasons:
-
Image not signed/attested by the expected attestor. The CI/CD pipeline did not execute the attestation step after building the image, or the attestation step failed silently. This is the most common cause - especially after adding a new attestor to the policy without updating the pipeline.
-
Wrong attestor key or key version. The attestor is configured with a KMS key that was rotated or deleted. The attestation was signed with an old key version, but the policy requires the current version. This happens after planned KMS key rotations without re-creating existing attestations.
-
Image digest mismatch - tag used instead of digest. Binary Authorization operates on immutable digests (sha256). If the deployment references an image by tag (e.g.,
:latest,:v1.2.3) but the attestation was created for a specific digest - verification fails because the tag may have been overwritten. -
Break-glass annotation missing for emergency deployments. In emergency situations (production hotfix, critical vulnerability) the deployment must include a special annotation that bypasses the Binary Authorization policy. Without it, even urgent fixes are blocked.
Solution
Step 1: Check the active Binary Authorization policy on the cluster
# Export the Binary Authorization policy for the project
gcloud container binauthz policy export --project=my-project
# Example output:
# admissionWhitelistPatterns:
# - namePattern: gcr.io/google_containers/*
# - namePattern: gcr.io/google-containers/*
# defaultAdmissionRule:
# enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
# evaluationMode: REQUIRE_ATTESTATION
# requireAttestationsBy:
# - projects/my-project/attestors/build-attestor
# - projects/my-project/attestors/security-scan-attestor
# clusterAdmissionRules:
# europe-west1-b.production-cluster:
# enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
# evaluationMode: REQUIRE_ATTESTATION
# requireAttestationsBy:
# - projects/my-project/attestors/build-attestor
# - projects/my-project/attestors/security-scan-attestor
# - projects/my-project/attestors/qa-attestor
# Check attestor configuration
gcloud container binauthz attestors describe build-attestor \
--project=my-project \
--format="yaml(name, userOwnedGrafeasNote, publicKeys)"
Step 2: Verify whether an attestation exists for the image
# Get the full image digest from the registry
gcloud container images describe gcr.io/my-project/my-app:v1.2.3 \
--format="value(image_summary.digest)"
# Output: sha256:a1b2c3d4e5f6...
# List attestations for this digest
gcloud container binauthz attestations list \
--project=my-project \
--attestor=build-attestor \
--attestor-project=my-project \
--artifact-url="gcr.io/my-project/my-app@sha256:a1b2c3d4e5f6..."
# If the list is empty - no attestation exists
# If attestation exists but deployment is still denied - check the key
# Check attestations from security-scan-attestor (if policy requires both)
gcloud container binauthz attestations list \
--project=my-project \
--attestor=security-scan-attestor \
--attestor-project=my-project \
--artifact-url="gcr.io/my-project/my-app@sha256:a1b2c3d4e5f6..."
Step 3: Create the missing attestation
# Get the attestor's public key ID
gcloud container binauthz attestors describe build-attestor \
--project=my-project \
--format="json(userOwnedGrafeasNote.publicKeys)"
# Create attestation using KMS key
gcloud container binauthz attestations sign-and-create \
--project=my-project \
--artifact-url="gcr.io/my-project/my-app@sha256:a1b2c3d4e5f6..." \
--attestor=build-attestor \
--attestor-project=my-project \
--keyversion-project=my-project \
--keyversion-location=europe-west1 \
--keyversion-keyring=binauthz-keys \
--keyversion-key=build-attestor-key \
--keyversion=1
# Confirm attestation was created
gcloud container binauthz attestations list \
--project=my-project \
--attestor=build-attestor \
--attestor-project=my-project \
--artifact-url="gcr.io/my-project/my-app@sha256:a1b2c3d4e5f6..."
# Expected: attestation visible in the list
Step 4: Fix image references - use digest instead of tag
# WRONG - deployment references a mutable tag
# The tag may point to a different digest than what was attested
spec:
containers:
- name: my-app
image: gcr.io/my-project/my-app:v1.2.3
# CORRECT - deployment references an immutable digest
spec:
containers:
- name: my-app
image: gcr.io/my-project/my-app@sha256:a1b2c3d4e5f6...
# Automatically replace tag with digest in your manifest
IMAGE_DIGEST=$(gcloud container images describe \
gcr.io/my-project/my-app:v1.2.3 \
--format="value(image_summary.fully_qualified_digest)")
sed -i "s|gcr.io/my-project/my-app:v1.2.3|${IMAGE_DIGEST}|g" deployment.yaml
# Apply the corrected manifest
kubectl apply -f deployment.yaml
Step 5: Emergency deployment with break-glass (critical situations only)
# Break-glass annotation bypasses Binary Authorization
# WARNING: generates an alert in Cloud Audit Logs - requires post-mortem
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app-emergency
annotations:
alpha.image-policy.k8s.io/break-glass: "true"
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
annotations:
alpha.image-policy.k8s.io/break-glass: "true"
spec:
containers:
- name: my-app
image: gcr.io/my-project/my-app:v1.2.3-hotfix
# Deploy with break-glass
kubectl apply -f deployment-emergency.yaml
# IMMEDIATELY after deployment: create attestation for the hotfix image
# and redeploy without break-glass (compliance requirement)
gcloud container binauthz attestations sign-and-create \
--project=my-project \
--artifact-url="gcr.io/my-project/my-app@sha256:hotfix-digest..." \
--attestor=build-attestor \
--attestor-project=my-project \
--keyversion-project=my-project \
--keyversion-location=europe-west1 \
--keyversion-keyring=binauthz-keys \
--keyversion-key=build-attestor-key \
--keyversion=1
Step 6: Fix the CI/CD pipeline - add attestation step
# Example attestation step in Cloud Build (cloudbuild.yaml)
steps:
# ... build and push steps ...
- id: 'create-attestation'
name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- '-c'
- |
IMAGE_DIGEST=$(gcloud container images describe \
gcr.io/${PROJECT_ID}/my-app:${SHORT_SHA} \
--format="value(image_summary.digest)")
gcloud container binauthz attestations sign-and-create \
--project=${PROJECT_ID} \
--artifact-url="gcr.io/${PROJECT_ID}/my-app@$${IMAGE_DIGEST}" \
--attestor=build-attestor \
--attestor-project=${PROJECT_ID} \
--keyversion-project=${PROJECT_ID} \
--keyversion-location=europe-west1 \
--keyversion-keyring=binauthz-keys \
--keyversion-key=build-attestor-key \
--keyversion=1
waitFor: ['push-image']
Step 7: Verify attestor key alignment after KMS rotation
If the denial started after a KMS key rotation, the attestor needs to be updated with the new key version:
# Check current key versions in the keyring
gcloud kms keys versions list \
--key=build-attestor-key \
--keyring=binauthz-keys \
--location=europe-west1 \
--project=my-project \
--format="table(name, state, createTime)"
# If key version 1 is DISABLED and version 2 is ENABLED,
# update the attestor to use the new key version
gcloud container binauthz attestors public-keys add \
--attestor=build-attestor \
--project=my-project \
--keyversion-project=my-project \
--keyversion-location=europe-west1 \
--keyversion-keyring=binauthz-keys \
--keyversion-key=build-attestor-key \
--keyversion=2
# Remove the old (disabled) key from the attestor
OLD_KEY_ID=$(gcloud container binauthz attestors describe build-attestor \
--project=my-project \
--format="value(userOwnedGrafeasNote.publicKeys[0].id)")
gcloud container binauthz attestors public-keys remove "${OLD_KEY_ID}" \
--attestor=build-attestor \
--project=my-project
# Re-attest images that were signed with the old key
# (only needed for images currently deployed or about to be deployed)
gcloud container binauthz attestations sign-and-create \
--project=my-project \
--artifact-url="gcr.io/my-project/my-app@sha256:a1b2c3d4e5f6..." \
--attestor=build-attestor \
--attestor-project=my-project \
--keyversion-project=my-project \
--keyversion-location=europe-west1 \
--keyversion-keyring=binauthz-keys \
--keyversion-key=build-attestor-key \
--keyversion=2
Validation
After fixing the attestation or pipeline, confirm that deployment succeeds:
# 1. Retry the deployment
kubectl apply -f deployment.yaml
# Expected: deployment.apps/my-app configured (no admission error)
# 2. Check pod status
kubectl get pods -l app=my-app
# Expected: STATUS = Running
# 3. Confirm no new denials in Audit Logs
gcloud logging read \
'resource.type="k8s_cluster" AND
protoPayload.methodName="io.k8s.core.v1.pods.create" AND
protoPayload.response.reason="VIOLATES_POLICY"' \
--project=my-project \
--freshness=10m \
--format="table(timestamp, protoPayload.response.reason)"
# Expected: no results (no new denials)
# 4. Check for break-glass events (should be 0 in normal operation)
gcloud logging read \
'resource.type="k8s_cluster" AND
protoPayload.request.metadata.annotations."alpha.image-policy.k8s.io/break-glass"="true"' \
--project=my-project \
--freshness=24h
# Expected: no results (break-glass used only in emergencies)
# 5. Verify attestor and KMS key are aligned
gcloud container binauthz attestors describe build-attestor \
--project=my-project \
--format="value(userOwnedGrafeasNote.publicKeys[0].id)"
# Compare with key version used in the pipeline
Binary Authorization is a critical supply chain security control required by PCI DSS and SOC2. A blocked deployment means the mechanism is working correctly - the problem is in the attestation process, not in the policy. Never disable Binary Authorization in production. Instead, fix the pipeline so every image is automatically attested after passing security scanning. Break-glass is strictly for emergencies, and every use must be documented in a post-mortem.
Binary Authorization blocking your deployments?
Book a free 30-minute call. We design CI/CD pipelines with automated image attestation, attestor configuration, and PCI DSS compliant policies.