Google Cloud Cross-Cloud Interconnect BGP session flapping: route advertisement limit and peering instability

Symptoms

Cloud Router logs show the BGP session oscillating:

BGP peer aws-peer on interface interconnect-attachment-0 changed state from ESTABLISHED to IDLE
BGP peer aws-peer on interface interconnect-attachment-0 changed state from IDLE to ESTABLISHED

These transitions repeat every 30-90 seconds. Routes advertised from the remote side appear and disappear in the VPC route table:

gcloud compute routes list --filter="network=my-vpc" --format="table(destRange, nextHopIp, priority)"

Shows routes flickering in and out between consecutive runs.

Cloud Logging reveals the reset reason:

Received BGP notification: cease/administrative reset

Or:

BGP session reset: hold timer expired

Cross-cloud connectivity is intermittent - some requests succeed, others timeout depending on whether routes are currently installed.

Cause

BGP session flapping on Cross-Cloud Interconnect typically stems from one of these issues:

1. Exceeded maximum advertised routes. Cloud Router has a default limit of 100 learned routes per BGP session. When the remote side (AWS) advertises more than 100 prefixes, Cloud Router resets the session to protect itself from route table overflow. The session re-establishes, learns routes again, hits the limit, and resets in a loop.

2. ASN conflict between Cloud Router and AWS side. If both sides accidentally use the same ASN (e.g., both using 65000), BGP detects an AS loop in received updates and rejects all routes. Some implementations reset the session when all routes are rejected.

3. Hold timer mismatch. GCP Cloud Router defaults to a 60-second hold timer (20-second keepalive). If the AWS side uses a shorter hold timer or if keepalives are delayed due to congestion on the interconnect, the hold timer expires and the session drops.

4. MTU mismatch on VLAN attachment. Cross-Cloud Interconnect VLAN attachments support up to 1500 bytes MTU for the BGP control plane. If the MTU is misconfigured (e.g., set to 1440 for IPsec compatibility), large BGP UPDATE messages carrying many routes get fragmented or dropped silently, causing the peer to think the session is dead.

5. BFD (Bidirectional Forwarding Detection) too sensitive. If BFD is enabled with aggressive timers (e.g., 300ms interval, 3x multiplier = 900ms detection), any brief congestion or microloop causes BFD to declare the link down and tear the BGP session, even though the link is physically fine.

Fix

Step 1: Check current BGP session status and route count

gcloud compute routers get-status my-cloud-router \
  --region us-east4 \
  --format="yaml(result.bgpPeerStatus)"

Look at numLearnedRoutes for each peer. If it is at or near 100, you have hit the route limit.

Step 2: Increase the learned route limit

If route count is the issue, increase the limit on the Cloud Router:

gcloud compute routers update-bgp-peer my-cloud-router \
  --peer-name aws-peer \
  --region us-east4 \
  --set-advertisement-mode custom \
  --set-advertisement-ranges 10.0.0.0/8

# Increase the route limit (requires updating the router)
gcloud compute routers update my-cloud-router \
  --region us-east4 \
  --set-bgp-peer-asn 65001 \
  --bgp-peer-name aws-peer

Alternatively, request a quota increase for the maximum number of learned routes:

gcloud compute routers update my-cloud-router \
  --region us-east4 \
  --set-peer-ip-address 169.254.10.2 \
  --set-peer-asn 65001

On the AWS side, use route summarization to reduce the number of advertised prefixes:

# On AWS Transit Gateway, configure summarization
aws ec2 create-transit-gateway-route \
  --transit-gateway-route-table-id tgw-rtb-0abc123 \
  --destination-cidr-block 10.0.0.0/8 \
  --transit-gateway-attachment-id tgw-attach-0def456

Step 3: Verify ASN configuration

Check Cloud Router ASN:

gcloud compute routers describe my-cloud-router \
  --region us-east4 \
  --format="value(bgp.asn)"

Check the peer ASN configured on Cloud Router:

gcloud compute routers describe my-cloud-router \
  --region us-east4 \
  --format="yaml(bgpPeers[].peerAsn)"

Verify on AWS side:

aws directconnect describe-virtual-interfaces \
  --query 'virtualInterfaces[?virtualInterfaceName==`my-cross-cloud-vif`].{ASN:asn,AmazonASN:amazonSideAsn}' \
  --output table

The Cloud Router ASN must differ from the AWS-side ASN. If they match, update one side:

# Change Cloud Router ASN (requires recreation)
gcloud compute routers delete my-cloud-router --region us-east4 --quiet

gcloud compute routers create my-cloud-router \
  --region us-east4 \
  --network my-vpc \
  --asn 65010

Step 4: Align hold timers

Check current hold timer on Cloud Router:

gcloud compute routers describe my-cloud-router \
  --region us-east4 \
  --format="yaml(bgp.keepaliveInterval)"

Set a consistent hold timer. GCP Cloud Router supports configuring the keepalive interval (hold timer = 3x keepalive):

gcloud compute routers update my-cloud-router \
  --region us-east4 \
  --keepalive-interval 20

This sets the hold timer to 60 seconds (3 x 20). Verify the AWS side uses a matching or longer hold timer:

aws directconnect describe-bgp-peers \
  --query 'bgpPeers[*].{Peer:bgpPeerId,State:bgpPeerState,ASN:asn}' \
  --output table

On the AWS router configuration, ensure the hold timer is at least 60 seconds (default in most AWS configurations).

Step 5: Check and fix VLAN attachment MTU

gcloud compute interconnects attachments describe my-attachment \
  --region us-east4 \
  --format="value(mtu)"

For Cross-Cloud Interconnect, the MTU should be 1440 or 1500 depending on the interconnect type. If BGP UPDATE messages are large (many routes), ensure MTU allows unfragmented delivery:

gcloud compute interconnects attachments update my-attachment \
  --region us-east4 \
  --mtu 1500

Test that BGP keepalives and updates pass without fragmentation:

# From a test instance on the interconnect subnet
ping -M do -s 1472 169.254.10.1

If this fails, reduce to 1440:

ping -M do -s 1412 169.254.10.1

Step 6: Tune or disable BFD

If BFD is causing premature session teardowns, increase the detection interval:

gcloud compute routers update-bgp-peer my-cloud-router \
  --peer-name aws-peer \
  --region us-east4 \
  --bfd-min-receive-interval 1000 \
  --bfd-min-transmit-interval 1000 \
  --bfd-multiplier 5

This sets BFD detection time to 5 seconds (1000ms x 5), which is tolerant of brief congestion while still detecting actual link failures faster than the BGP hold timer.

To disable BFD entirely during troubleshooting:

gcloud compute routers update-bgp-peer my-cloud-router \
  --peer-name aws-peer \
  --region us-east4 \
  --bfd-session-initialization-mode disabled

Step 7: Monitor Cloud Router logs for remaining issues

gcloud logging read '
  resource.type="gce_router"
  AND resource.labels.router_id="my-cloud-router"
  AND severity>=WARNING
  AND timestamp>="2026-07-14T00:00:00Z"
' --project my-project --limit 50 --format="table(timestamp, severity, textPayload)"

Validation

Confirm BGP session is stable for 10+ minutes:

gcloud compute routers get-status my-cloud-router \
  --region us-east4 \
  --format="yaml(result.bgpPeerStatus[].{name:name,status:status,uptimeSeconds:uptimeSeconds,numLearnedRoutes:numLearnedRoutes})"

Expected output:

- name: aws-peer
  status: UP
  uptimeSeconds: '600'
  numLearnedRoutes: 45

Verify route count is consistent and below the limit:

gcloud compute routes list \
  --filter="network=my-vpc AND nextHopIp:169.254.*" \
  --format="table(destRange, nextHopIp, priority)" | wc -l

Run this command 5 times over 5 minutes - the count should remain the same.

Check Cloud Router logs for any flaps in the last 15 minutes:

gcloud logging read '
  resource.type="gce_router"
  AND textPayload:"changed state"
  AND timestamp>="2026-07-14T08:45:00Z"
' --project my-project --limit 10

Expected: no results (no state changes).

Test cross-cloud connectivity end-to-end:

# From GCP instance to AWS instance
for i in $(seq 1 10); do
  ping -c 1 -W 2 10.0.1.50 && echo "OK" || echo "FAIL"
  sleep 5
done

All 10 attempts should return OK with no intermittent failures.

Need help with multi-cloud networking?

Book a free 30-minute call. We will diagnose your connectivity issue and recommend the right fix.

Book a call