AWS VPN BGP Transit Gateway routing networking GCP

AWS Site-to-Site VPN with multiple connections: fixing asymmetric routing and traffic hairpinning

Fix asymmetric routing and traffic hairpinning when running multiple AWS Site-to-Site VPN connections in parallel to GCP Cloud VPN.

Jerzy Kopaczewski · 2026-06-05

You added a second or third VPN connection for bandwidth, but now traffic takes unpredictable paths. Some requests go out one tunnel and responses come back on another. Stateful firewalls drop these asymmetric flows, causing intermittent timeouts. This runbook shows how to control routing across parallel VPN connections.

Symptoms

After adding multiple VPN connections between AWS and GCP, intermittent connectivity issues appear:

Latency spikes for specific subnets while others are fine:

# Some destinations have stable latency
ping -c 10 10.128.1.10
# rtt min/avg/max = 22.1/23.4/24.8 ms

# Others spike unpredictably
ping -c 10 10.128.2.10
# rtt min/avg/max = 22.5/89.3/245.1 ms

Stateful firewall (Security Groups, NACLs, or GCP firewall) drops packets:

# VPC Flow Logs show REJECT for return traffic
aws logs filter-log-events \
  --log-group-name /aws/vpc/flowlogs \
  --filter-pattern "REJECT" \
  --start-time $(date -u -d '1 hour ago' +%s)000

2 123456789012 eni-0abc123 10.128.2.10 10.0.1.50 443 52431 6 5 300 1720000000 1720000060 REJECT OK

Requests timeout intermittently - typically 10-30% failure rate:

for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://10.128.2.10:8080/health
  sleep 1
done

0.045s
0.044s
30.001s  # timeout
0.046s
30.001s  # timeout
0.043s

Transit Gateway route table shows equal-cost paths:

aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id tgw-rtb-0abc123 \
  --filters "Name=route-search.exact-match,Values=10.128.0.0/16" \
  --output table

CIDR            Type     State    Attachments
10.128.0.0/16   propagated  active  tgw-attach-vpn1, tgw-attach-vpn2, tgw-attach-vpn3

Cause

Asymmetric routing with parallel VPN connections happens because of routing control issues:

BGP MED/local preference not aligned between connections. When multiple VPN connections advertise the same prefixes with equal MED (Multi-Exit Discriminator) values, the Transit Gateway has no preference and load-balances across all connections. Request and response packets may take different paths.
ECMP load balancing distributing request/response across different tunnels. AWS Transit Gateway uses ECMP (Equal-Cost Multi-Path) by default. The hash is based on source/destination IP and port. When the hash differs between outbound and inbound packets (common with NAT or port changes), traffic splits across tunnels.
Transit Gateway route table has equal-cost paths without preference. All VPN attachments propagate the same routes with identical metrics. The Transit Gateway treats them as equal and distributes flows without stickiness.
GCP Cloud Router advertising same prefixes on all tunnels without AS path prepending. If GCP advertises 10.128.0.0/16 identically on all VPN tunnels, the AWS side sees multiple equal paths and cannot establish a primary/backup hierarchy.
Security groups/NACLs seeing traffic from unexpected source tunnel IP. When return traffic comes through a different tunnel, the source IP (tunnel endpoint) differs from what the stateful tracking expects. Connection tracking on security groups may reject these as invalid.

Fix

Step 1: Define primary and backup connections

Decide which VPN connection(s) are primary and which are backup. For example:

vpn-connection-1 (via tunnel endpoint 203.0.113.1) = primary
vpn-connection-2 (via tunnel endpoint 203.0.113.2) = secondary
vpn-connection-3 (via tunnel endpoint 203.0.113.3) = tertiary/backup

Step 2: Set BGP MED values on GCP Cloud Router

Configure different MED values per tunnel on GCP Cloud Router. Lower MED = higher preference:

# Primary tunnel - lowest MED (most preferred)
gcloud compute routers update-bgp-peer my-cloud-router \
  --peer-name aws-primary \
  --region us-central1 \
  --advertisement-mode custom \
  --set-advertisement-ranges "10.128.0.0/16=100"

# Secondary tunnel - higher MED
gcloud compute routers update-bgp-peer my-cloud-router \
  --peer-name aws-secondary \
  --region us-central1 \
  --advertisement-mode custom \
  --set-advertisement-ranges "10.128.0.0/16=200"

# Backup tunnel - highest MED
gcloud compute routers update-bgp-peer my-cloud-router \
  --peer-name aws-backup \
  --region us-central1 \
  --advertisement-mode custom \
  --set-advertisement-ranges "10.128.0.0/16=300"

Step 3: Configure AS path prepending on backup tunnels

AS path prepending is a more universally respected BGP attribute than MED. Prepend the AS path on backup tunnels to make them less preferred:

On GCP Cloud Router, use route policies to prepend:

# Create a route policy for the backup peer
gcloud compute routers add-route-policy my-cloud-router \
  --region us-central1 \
  --policy-name backup-prepend \
  --policy-type export \
  --terms '[{
    "priority": 1,
    "match": {"prefixes": [{"prefix": "10.128.0.0/16"}]},
    "actions": {"prependAsPath": ["65010", "65010", "65010"]}
  }]'

# Apply the policy to the backup peer
gcloud compute routers update-bgp-peer my-cloud-router \
  --peer-name aws-backup \
  --region us-central1 \
  --export-policies backup-prepend

This makes the backup path appear 3 AS hops longer, ensuring AWS always prefers the primary and secondary paths.

Step 4: Verify Transit Gateway route preferences

Check how routes are being installed in the Transit Gateway route table:

aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id tgw-rtb-0abc123 \
  --filters "Name=route-search.exact-match,Values=10.128.0.0/16" \
  --query 'Routes[*].{CIDR:DestinationCidrBlock,Attachments:TransitGatewayAttachments[*].TransitGatewayAttachmentId,Type:Type}' \
  --output json

If ECMP is not desired, disable it on the Transit Gateway:

aws ec2 modify-transit-gateway \
  --transit-gateway-id tgw-0abc123def456 \
  --options "VpnEcmpSupport=disable"

With ECMP disabled, the Transit Gateway uses only the best path based on BGP attributes (AS path length, MED, origin).

If you want ECMP for bandwidth but symmetric routing for sessions, keep ECMP enabled but ensure both directions (AWS to GCP and GCP to AWS) use the same path selection criteria.

Step 5: Set AWS-side route preferences

On the AWS VPN configuration, set different local preference for routes learned from each connection. This is done via Transit Gateway route table static routes or BGP manipulation on the customer gateway:

# Add a static route with higher priority pointing to primary VPN
aws ec2 create-transit-gateway-route \
  --transit-gateway-route-table-id tgw-rtb-0abc123 \
  --destination-cidr-block 10.128.0.0/16 \
  --transit-gateway-attachment-id tgw-attach-vpn1

For BGP-based preference, configure the VPN tunnel options to prefer one tunnel:

aws ec2 modify-vpn-tunnel-options \
  --vpn-connection-id vpn-0abc123def456 \
  --vpn-tunnel-outside-ip-address 203.0.113.1 \
  --tunnel-options '{
    "Phase1EncryptionAlgorithms": [{"Value": "AES256"}],
    "Phase2EncryptionAlgorithms": [{"Value": "AES256-GCM-16"}]
  }'

Step 6: Configure GCP custom route advertisements per tunnel

Instead of advertising the same /16 on all tunnels, split prefixes across tunnels for deterministic routing:

# Primary tunnel advertises all subnets
gcloud compute routers update-bgp-peer my-cloud-router \
  --peer-name aws-primary \
  --region us-central1 \
  --advertisement-mode custom \
  --set-advertisement-ranges "10.128.0.0/16"

# Secondary tunnel advertises only specific subnets (for failover)
gcloud compute routers update-bgp-peer my-cloud-router \
  --peer-name aws-secondary \
  --region us-central1 \
  --advertisement-mode custom \
  --set-advertisement-ranges "10.128.1.0/24,10.128.2.0/24"

This ensures that the more-specific routes (/24) are only available via specific tunnels, preventing asymmetric paths for those subnets.

Step 7: Fix stateful firewall rules

If asymmetric routing cannot be completely eliminated (e.g., during failover), ensure security groups and NACLs allow traffic from all tunnel endpoint IPs:

# Allow return traffic from all VPN tunnel endpoints
aws ec2 authorize-security-group-ingress \
  --group-id sg-0abc123 \
  --protocol -1 \
  --cidr 10.128.0.0/16

# Check NACL rules allow traffic in both directions
aws ec2 describe-network-acls \
  --filters "Name=association.subnet-id,Values=subnet-0abc123" \
  --query 'NetworkAcls[0].Entries[*].{Rule:RuleNumber,CIDR:CidrBlock,Action:RuleAction,Egress:Egress}' \
  --output table

On GCP, ensure firewall rules accept traffic regardless of which tunnel it arrives on:

gcloud compute firewall-rules create allow-aws-all-tunnels \
  --network my-vpc \
  --allow all \
  --source-ranges 10.0.0.0/8 \
  --direction INGRESS \
  --priority 1000

Step 8: Verify route table state

Confirm the Transit Gateway routes show proper preference:

aws ec2 get-transit-gateway-route-table-associations \
  --transit-gateway-route-table-id tgw-rtb-0abc123 \
  --output table

aws ec2 get-transit-gateway-route-table-propagations \
  --transit-gateway-route-table-id tgw-rtb-0abc123 \
  --output table

Check BGP received routes on GCP Cloud Router:

gcloud compute routers get-status my-cloud-router \
  --region us-central1 \
  --format="yaml(result.bgpPeerStatus[].{name:name,numLearnedRoutes:numLearnedRoutes,advertisedRoutes:advertisedRoutes})"

Validation

Verify traceroute shows a consistent path:

# Run 5 times, path should be identical each time
for i in $(seq 1 5); do
  echo "--- Attempt $i ---"
  traceroute -n 10.128.2.10
  sleep 5
done

All attempts should show the same intermediate hops (same tunnel path).

Test latency stability:

ping -c 100 10.128.2.10 | tail -1

Expected: consistent avg with low standard deviation (e.g., rtt min/avg/max/mdev = 22.1/23.4/24.8/0.9 ms).

Verify no stateful drops:

# Check VPC Flow Logs for REJECTs to/from GCP subnets
aws logs filter-log-events \
  --log-group-name /aws/vpc/flowlogs \
  --filter-pattern "10.128 REJECT" \
  --start-time $(date -u -d '15 minutes ago' +%s)000 \
  --query 'events[*].message' \
  --output text

Expected: no results (no rejected traffic).

Run application-level connectivity test:

# 100 requests with no timeouts expected
for i in $(seq 1 100); do
  code=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 5 http://10.128.2.10:8080/health)
  if [ "$code" != "200" ]; then
    echo "FAIL at attempt $i: HTTP $code"
  fi
done
echo "Test complete"

Expected: all 100 requests return 200, zero failures.

Verify traffic distribution matches intent:

# Check CloudWatch metrics per VPN connection
for vpn in vpn-0abc123 vpn-0def456 vpn-0ghi789; do
  echo "--- $vpn ---"
  aws cloudwatch get-metric-statistics \
    --namespace "AWS/VPN" \
    --metric-name TunnelDataOut \
    --dimensions Name=VpnId,Value=$vpn \
    --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
    --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
    --period 3600 \
    --statistics Sum \
    --query 'Datapoints[0].Sum'
done

Primary should carry most traffic, secondary less, backup near zero (unless primary fails).

Need help with multi-cloud networking?

Book a free 30-minute call. We will diagnose your connectivity issue and recommend the right fix.