AWS Site-to-Site VPN throughput stuck below 1 Gbps: MTU/MSS clamping and IPsec fragmentation

Symptoms

iperf3 tests show throughput well below the tunnel’s 1.25 Gbps rating:

iperf3 -c 10.128.0.2 -t 30 -P 1

[  5]   0.00-30.00  sec  1.42 GBytes   407 Mbits/sec    sender
[  5]   0.00-30.00  sec  1.40 GBytes   401 Mbits/sec    receiver

Large file transfers (SCP, rsync) stall periodically or complete much slower than expected. TCP retransmits spike during bulk transfers:

ss -ti | grep -A 5 "10.128.0.2"

cubic wscale:7,7 rto:204 rtt:25.3/2.1 ato:40 mss:1360 pmtu:1500
retrans:0/847 rcv_space:65536

The high retransmit count (847 in this example) confirms fragmentation-related drops.

Small packets work perfectly - SSH sessions are responsive, API calls return quickly, DNS resolves without delay. Only bulk data transfers suffer.

CloudWatch VPN metrics confirm the tunnel is up but data transfer is suboptimal:

aws cloudwatch get-metric-statistics \
  --namespace "AWS/VPN" \
  --metric-name TunnelDataOut \
  --dimensions Name=VpnId,Value=vpn-0abc123def456 \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Sum

Cause

Throughput degradation on IPsec tunnels comes down to packet size and fragmentation:

1. Path MTU too high - IPsec overhead causes fragmentation. Standard Ethernet MTU is 1500 bytes. IPsec encapsulation adds 50-73 bytes of overhead (ESP header, IV, padding, authentication tag). When instances send 1500-byte packets, the VPN gateway must fragment them to fit within the tunnel. Fragmentation at the tunnel endpoint is CPU-intensive and halves effective throughput because each original packet becomes two.

2. MSS clamping not configured on VPN gateway. Without MSS clamping, TCP endpoints negotiate the maximum segment size based on their local interface MTU (1460 bytes for 1500 MTU). They send segments too large for the tunnel, causing fragmentation at the VPN gateway rather than smaller segments from the source.

3. DF (Don’t Fragment) bit set causing silent drops. Many applications set the DF bit on packets. When these packets are too large for the tunnel and fragmentation is not allowed, they are silently dropped. The sending host only learns about this when ICMP “Fragmentation Needed” messages make it back - which may be blocked by firewalls.

4. TCP window scaling disabled. Without window scaling, TCP cannot fill the bandwidth-delay product of the tunnel. Even with correct MTU settings, throughput stays low because the sender pauses waiting for ACKs.

5. Single-flow limitation. AWS VPN uses per-flow ECMP across tunnel paths. A single TCP stream is pinned to one tunnel and cannot exceed approximately 1.25 Gbps. If you are testing with a single iperf3 stream and expecting more, this is the ceiling.

Fix

Step 1: Set instance MTU to 1400

On EC2 instances that communicate across the VPN, reduce the interface MTU to account for IPsec overhead:

# On Amazon Linux / RHEL
sudo ip link set dev eth0 mtu 1400

# Make persistent via netplan (Ubuntu)
sudo tee /etc/netplan/99-mtu.yaml <<EOF
network:
  version: 2
  ethernets:
    eth0:
      mtu: 1400
EOF
sudo netplan apply

On GCP instances:

# GCP VPC MTU is set at the network level
gcloud compute networks update my-vpc --mtu 1400

Note: Changing GCP VPC MTU requires stopping and starting instances in that network.

Alternatively, set MTU per-instance on GCP:

sudo ip link set dev ens4 mtu 1400

Step 2: Configure MSS clamping

MSS clamping tells TCP to use a smaller segment size without changing the interface MTU. This is the preferred approach because it only affects TCP traffic traversing the tunnel.

On the AWS VPN, MSS clamping is configured in the tunnel options:

aws ec2 modify-vpn-tunnel-options \
  --vpn-connection-id vpn-0abc123def456 \
  --vpn-tunnel-outside-ip-address 203.0.113.1 \
  --tunnel-options "TunnelInsideIpVersion=ipv4"

AWS Site-to-Site VPN automatically applies MSS clamping to 1379 bytes. Verify with:

aws ec2 describe-vpn-connections \
  --vpn-connection-ids vpn-0abc123def456 \
  --query 'VpnConnections[0].Options.TunnelOptions[0]' \
  --output json | grep -i mss

On Linux instances, you can add iptables-based MSS clamping as a safety net:

sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --set-mss 1360

# Make persistent
sudo iptables-save | sudo tee /etc/iptables/rules.v4

On GCP Cloud Router, MSS clamping is not directly configurable but the VPN gateway handles it automatically when tunnel MTU is set correctly.

Step 3: Verify Path MTU Discovery (PMTUD) works

Test that ICMP “Fragmentation Needed” messages are not being blocked:

# From EC2 instance, test PMTUD to GCP instance
tracepath -n 10.128.0.2

Expected output shows MTU discovery:

 1?: [LOCALHOST]                      pmtu 1500
 1:  10.0.1.1                           0.5ms pmtu 1400
 2:  169.254.10.1                       5.2ms pmtu 1400
 3:  10.128.0.2                        25.1ms reached
     Resume: pmtu 1400 hops 3 back 3

If PMTUD fails (no pmtu reduction shown), check security groups and NACLs:

# Ensure ICMP type 3 (Destination Unreachable) is allowed inbound
aws ec2 describe-security-groups \
  --group-ids sg-0abc123 \
  --query 'SecurityGroups[0].IpPermissions[?IpProtocol==`icmp`]'

Add ICMP type 3 if missing:

aws ec2 authorize-security-group-ingress \
  --group-id sg-0abc123 \
  --protocol icmp \
  --port -1 \
  --cidr 10.128.0.0/16

On GCP, ensure the firewall allows ICMP:

gcloud compute firewall-rules create allow-icmp-pmtud \
  --network my-vpc \
  --allow icmp \
  --source-ranges 10.0.0.0/8 \
  --direction INGRESS

Step 4: Test with different packet sizes

Run iperf3 with explicit MSS to identify the optimal segment size:

# Test with MSS 1360 (accounting for IPsec overhead)
iperf3 -c 10.128.0.2 -t 10 -M 1360

# Compare with default (likely 1460)
iperf3 -c 10.128.0.2 -t 10

# Test UDP to isolate from TCP behavior
iperf3 -c 10.128.0.2 -t 10 -u -b 1G -l 1360

If 1360 MSS gives significantly better throughput than default, the fix is confirmed.

Step 5: Enable TCP window scaling

Verify window scaling is enabled on instances:

sysctl net.ipv4.tcp_window_scaling

If disabled, enable it:

sudo sysctl -w net.ipv4.tcp_window_scaling=1
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

# Make persistent
echo "net.ipv4.tcp_window_scaling = 1" | sudo tee -a /etc/sysctl.d/99-vpn-tuning.conf
echo "net.ipv4.tcp_rmem = 4096 87380 16777216" | sudo tee -a /etc/sysctl.d/99-vpn-tuning.conf
echo "net.ipv4.tcp_wmem = 4096 65536 16777216" | sudo tee -a /etc/sysctl.d/99-vpn-tuning.conf
sudo sysctl --system

Step 6: Test multi-flow throughput

If single-stream throughput is around 1.1-1.25 Gbps (the per-tunnel limit), use parallel flows to saturate multiple tunnels:

# 4 parallel streams
iperf3 -c 10.128.0.2 -t 30 -P 4

If multi-stream shows significant improvement, the bottleneck is the per-flow ECMP limitation, not fragmentation. In this case, ensure your workload naturally produces multiple TCP flows (which most production applications do).

Step 7: Monitor for TCP retransmits after fix

# Watch retransmits in real-time
watch -n 1 'ss -ti dst 10.128.0.0/16 | grep retrans'

Retransmit counts should stay at zero or near-zero during bulk transfers.

Validation

Run a comprehensive throughput test after applying fixes:

iperf3 -c 10.128.0.2 -t 60 -P 1 --json | jq '.end.sum_sent.bits_per_second / 1000000'

Expected: 1100+ Mbps for a single stream.

Verify no fragmentation is occurring:

# Check for fragmentation on the instance
cat /proc/net/snmp | grep -A 1 "Ip:"

Look for FragCreates and FragFails - these should not be incrementing during transfers.

Test a real-world workload:

# Large file transfer
time scp -o "Compression=no" /tmp/testfile-1G user@10.128.0.2:/tmp/

# Expected: ~8-9 seconds for 1 GB at 1 Gbps+

Run tracepath to confirm PMTUD is working:

tracepath -n 10.128.0.2

The output should show pmtu 1400 (or whatever you configured) without asymm or !H markers.

Check that TCP retransmits are negligible during a sustained transfer:

# Start a transfer in background
iperf3 -c 10.128.0.2 -t 60 &

# Monitor retransmits
ss -ti dst 10.128.0.2 | grep retrans

Expected: retrans:0/0 or very low single-digit values.

Need help with multi-cloud networking?

Book a free 30-minute call. We will diagnose your connectivity issue and recommend the right fix.

Book a call