Terraform - Detecting State Drift and Safely Force-Unlocking a Locked State
Diagnosing and fixing Terraform state drift: reconciling manual changes, terraform import, refresh-only plan. Safe force-unlock of a state locked in DynamoDB.
terraform plan shows unexpected changes: resources to delete or modify that nobody planned. Alternatively: terraform plan or apply returns an error Error acquiring the state lock - the state is locked by an interrupted CI/CD process or a crashed terminal. Both problems block further IaC operations on the infrastructure.
This runbook covers diagnosing state drift and safely unlocking Terraform state. For a comparison of IaC tools (Terraform vs Pulumi), see the article Terraform vs Pulumi - which IaC tool fits your team. A basic runbook for simple state unlocking is Terraform State Lock Stuck. If your drift is specifically caused by manual console changes (ClickOps), see our dedicated Terraform drift from console changes runbook. For help with infrastructure as code management - book a consultation.
Symptom
Scenario A: State Drift
terraform plan shows changes that do not correspond to any modification in the code:
# Typical drift symptoms:
# 1. Plan wants to destroy resources that exist in the cloud
# "Plan: 0 to add, 0 to change, 3 to destroy."
# (but nobody removed these resources from code!)
# 2. Plan wants to change attributes despite no changes in .tf
# ~ resource "aws_security_group" "main" {
# ~ ingress = [...] -> [...] # "forces replacement"
# }
# 3. Plan shows "resource has been changed outside of Terraform"
# Note: Objects have changed outside of Terraform
# Check what the plan proposes
terraform plan -detailed-exitcode
# Exit code 0 = no changes (OK)
# Exit code 1 = error
# Exit code 2 = changes present (DRIFT!)
# Check specific differences
terraform plan -json | jq '.resource_changes[] | select(.change.actions != ["no-op"]) | {address, actions: .change.actions}'
# Compare state with reality
terraform show -json | jq '.values.root_module.resources[] | {type, name, values: (.values | keys)}'
Scenario B: State Lock Stuck
# Typical error message:
# Error: Error acquiring the state lock
#
# Error message: ConditionalCheckFailedException: The conditional request failed
# Lock Info:
# ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890
# Path: my-project/terraform.tfstate
# Operation: OperationTypePlan
# Who: user@hostname
# Version: 1.9.0
# Created: 2026-08-07 14:23:45.123456 +0000 UTC
# Check who holds the lock in DynamoDB
aws dynamodb get-item \
--table-name terraform-locks \
--key '{"LockID": {"S": "my-bucket/my-project/terraform.tfstate"}}' \
--query 'Item.Info.S' --output text | jq .
# Shows: Who, Operation, Created timestamp
# Check if the CI/CD process is still running
# If Created > 30 min ago and pipeline is not active → lock is orphaned
Root Cause
State Drift - why state does not match infrastructure:
1. Manual changes in console/CLI (ClickOps):
Someone changed a security group, added a tag, modified a parameter through the AWS/Azure/GCP console. Terraform state does not know about these changes - it shows the “old” picture of the infrastructure.
2. Another IaC process manages the same resources:
A CloudFormation stack, Ansible playbook, Pulumi program, or another Terraform workspace modifies resources that are also in this state. Shared resource conflicts.
3. Auto-scaling / AWS default behaviours:
AWS automatically adds tags, modifies security groups (e.g. VPC default SG), changes attributes of managed resources. Terraform sees these changes as drift.
4. Corrupted or unsynchronised state:
An interrupted terraform apply saved partial state. Resources were created, but state was not updated. Or S3 versioning restored an older version of the state.
State Lock - why the lock is “stuck”:
1. CI/CD pipeline interrupted during operation:
GitHub Actions job cancelled, Jenkins node crashed, terminal closed during terraform apply. The process did not have time to release the lock in DynamoDB.
2. Network timeout to DynamoDB:
The Terraform operation completed, but the unlock request to DynamoDB did not arrive (network timeout, IAM temporary credentials expired).
3. Long apply without timeout:
terraform apply on large infrastructure (>100 resources) runs for hours. The lock is active, but another developer tries to plan on the same state.
Solution
Drift - reconcile state with reality:
A) Refresh state (update state without changing infrastructure):
# 1. First make a state backup!
terraform state pull > terraform.tfstate.backup.$(date +%Y%m%d_%H%M%S)
# 2. Refresh - synchronise state with current infrastructure
terraform plan -refresh-only
# Check what will change in state (not in infrastructure!)
# Shows: "Terraform will update the state to reflect changes made outside"
# 3. If changes look OK - apply the refresh
terraform apply -refresh-only -auto-approve
# 4. Now a normal plan should be clean
terraform plan
# Expected: "No changes. Your infrastructure matches the configuration."
B) Import resources that exist but are not in state:
# Scenario: resource exists in cloud, but Terraform does not know about it
# Plan shows: "resource will be created" - even though it already exists
# 1. Check if resource exists
aws ec2 describe-security-groups --group-ids sg-0123456789abcdef
# 2. Import into state (Terraform 1.5+ - import block in code)
# In your .tf file add:
# imports.tf - declarative import (Terraform 1.5+)
import {
to = aws_security_group.main
id = "sg-0123456789abcdef"
}
# Or for resources with composite ID:
import {
to = aws_route_table_association.private
id = "subnet-abc123/rtbassoc-xyz789"
}
# 3. Run plan - Terraform will compare config with the imported resource
terraform plan -generate-config-out=generated.tf
# generated.tf contains configuration matching the imported resource
# 4. Move generated configuration to the proper .tf file
# Adjust to your style/conventions
# 5. Run apply (import + alignment)
terraform apply
# 6. Remove import {} blocks after successful import
C) State rm - remove resource from state (not from infrastructure):
# Scenario: Terraform wants to delete a resource that should exist
# but should not be managed by THIS workspace
# 1. Check what is in state
terraform state list | grep "resource_to_remove"
# 2. Remove from state (resource stays in infrastructure!)
terraform state rm 'aws_security_group.legacy'
# "Successfully removed 1 resource instance(s)."
# 3. Verify - plan should not try to delete it
terraform plan
D) Moved blocks - refactoring without drift:
# When you renamed/moved resources in code and plan wants to destroy+create:
moved {
from = aws_instance.web_server
to = aws_instance.app_server
}
moved {
from = module.old_name
to = module.new_name
}
State Lock - safe unlocking:
E) Force-unlock (after verification!):
# IMPORTANT: NEVER force-unlock if another process is still running!
# Always check first:
# 1. Check who holds the lock
aws dynamodb get-item \
--table-name terraform-locks \
--key '{"LockID": {"S": "my-bucket/my-project/terraform.tfstate"}}' \
--query 'Item.Info.S' --output text | jq .
# 2. Verify the process is still alive
# - Check CI/CD pipeline (is the job active?)
# - Check the Who field - does that person still have a terminal open?
# - Check Created timestamp - if > 1h ago, likely orphaned
# 3. If the lock is orphaned - unlock it
LOCK_ID="a1b2c3d4-e5f6-7890-abcd-ef1234567890" # from the error message
terraform force-unlock $LOCK_ID
# 4. If force-unlock does not work (state corrupted) - delete manually from DynamoDB
aws dynamodb delete-item \
--table-name terraform-locks \
--key '{"LockID": {"S": "my-bucket/my-project/terraform.tfstate"}}'
# 5. IMMEDIATELY after unlock - run plan to verify state
terraform plan
F) Preventing future stuck locks in CI/CD:
# GitHub Actions - timeout + cleanup
name: Terraform Apply
on: [push]
jobs:
apply:
runs-on: ubuntu-latest
timeout-minutes: 60 # Hard limit on the entire job
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- name: Terraform Apply
id: apply
timeout-minutes: 45 # Timeout on apply (leaves time for unlock)
run: terraform apply -auto-approve
continue-on-error: true
# Cleanup lock if apply failed/timed out
- name: Force Unlock on Failure
if: failure() || cancelled()
run: |
LOCK_ID=$(terraform force-unlock -force 2>&1 | grep -oP '[a-f0-9-]{36}' | head -1)
if [ -n "$LOCK_ID" ]; then
terraform force-unlock -force "$LOCK_ID"
fi
# backend.tf - configuration with lock timeout
terraform {
backend "s3" {
bucket = "my-terraform-state"
key = "my-project/terraform.tfstate"
region = "eu-west-1"
dynamodb_table = "terraform-locks"
encrypt = true
# Timeout on lock acquisition (default: 0 = wait forever)
# Set shorter than CI/CD job timeout
skip_metadata_api_check = false
}
}
Validation
# 1. After fixing drift - plan must be clean
terraform plan -detailed-exitcode
# Expected: exit code 0 (no changes)
# 2. After force-unlock - plan must execute without lock errors
terraform plan
# Expected: no "Error acquiring the state lock"
# 3. Verify state is consistent
terraform state list | wc -l
# Compare with expected number of resources
# 4. Check there are no "tainted" resources
terraform show -json | jq '[.values.root_module.resources[] | select(.tainted == true)] | length'
# Expected: 0
# 5. Drift detection - configure automatic detection
terraform plan -refresh-only -detailed-exitcode
# In CI/CD: run scheduled job every 24h
# Exit code 2 = drift detected → alert to the team
# 6. Check DynamoDB - lock should be free
aws dynamodb get-item \
--table-name terraform-locks \
--key '{"LockID": {"S": "my-bucket/my-project/terraform.tfstate"}}' \
--query 'Item'
# Expected: no Item (lock released) or null
Terraform state causing problems?
Schedule a free 30-minute call. We will help fix drift, configure automatic detection, and secure your CI/CD pipeline against stuck locks.