Logo
AWS AWS MGN VMware migration rehosting

AWS MGN: replication stuck, lag rising, test-launch won't boot

Fix stalled AWS MGN replication: diagnose agent and source-server state, network bandwidth and throughput, test-instance boot failures, and replication-server sizing.

Jerzy Kopaczewski ·
AWS MGN replication was meant to get your servers ready before the cutover window, but progress is stuck at X%, lag is rising instead of falling, or the agent on the source machine keeps disconnecting. Sometimes replication shows "Healthy" but the test-launch produces a machine that won't boot. This runbook shows how to diagnose the cause and get replication to a cutover-ready state.

This runbook covers troubleshooting AWS MGN (Application Migration Service) replication during a VMware to AWS migration. For the full guide to the migration itself, see VMware to AWS Migration - Complete Guide. For migration planning and delivery, book a consulting session.

Symptoms

In the MGN console the source server sits in the “Initial sync” phase and never reaches 100%, or drops back into resync. Typical signals:

# Source servers and their replication phase (data replication)
aws mgn describe-source-servers \
  --query 'items[].{host:sourceProperties.identificationHints.hostname, state:dataReplicationInfo.dataReplicationState, backlog:dataReplicationInfo.replicatedDisks[].backlogedStorageBytes, lag:dataReplicationInfo.lagDuration}' \
  --output table

# Typical dataReplicationState values:
#   INITIAL_SYNC        - first sync in progress
#   STALLED             - replication stuck (the most common problem)
#   RESCAN              - MGN detected a divergence and is re-scanning disks
#   CONTINUOUS          - target state (cutover-ready)

# Messages visible in the console / events:
# "Data replication has stalled"
# "Agent has not reported for X minutes"
# "Not enough free disk space on the replication server"

dataReplicationState = STALLED, or lagDuration grows over time instead of trending to zero. In the worst case the server cycles into RESCAN repeatedly.

Cause

Stalled MGN replication and rising lag usually have one of four causes:

  • Network bandwidth source → AWS: the replication agent ships blocks over port 1500/TCP to the replication server in the VPC. A thin outbound link from on-prem, insufficient Direct Connect/VPN capacity, or throttling on the firewall causes the backlog to grow faster than MGN can catch up. This is the most common cause of rising lag when migrating large volumes.
  • Replication server too small or out of disk: MGN runs lightweight replication instances (t3.small by default) with staging volumes in the target account. With many parallel source servers or large disks it runs out of IOPS or staging space - replication drops into STALLED with an out-of-space message.
  • Agent disconnects / not reporting: a blocked port 443 to the MGN endpoint, expired credentials used when installing the agent, a source-machine reboot, or antivirus blocking the agent process cause MGN to lose contact with the server and pause replication.
  • Rescan after a source-side change: a large change on the source disks (defragmentation, VMware snapshot, disk resize) forces a re-scan. Repeated RESCANs usually mean unstable source disks or concurrent operations running against the VM.

Fix

A) Determine whether the bottleneck is the network or the replication server:

# Backlog and lag per source server - is the backlog shrinking over time?
aws mgn describe-source-servers \
  --query 'items[].{host:sourceProperties.identificationHints.hostname, state:dataReplicationInfo.dataReplicationState, lag:dataReplicationInfo.lagDuration, etaDateTime:dataReplicationInfo.etaDateTime}' \
  --output table

# If the backlog is flat or growing -> network or replication server.
# Check throughput from the source machine to the replication endpoint (port 1500):
#   iperf3 / nc to the replication server IP in the VPC
# and drops on the on-prem firewall for ports 443 (MGN API) and 1500 (data).

B) Network bottleneck - throttling and the replication window:

# Throttle throughput per server so you don't saturate the production link during the day,
# and let it catch up overnight (set in the replication template).
aws mgn update-replication-configuration \
  --source-server-id s-1234567890abcdef0 \
  --bandwidth-throttling 500   # Mbps; 0 = unlimited

# Use a private path (Direct Connect / VPN) instead of public,
# to avoid limits and internet transfer costs:
aws mgn update-replication-configuration \
  --source-server-id s-1234567890abcdef0 \
  --use-dedicated-replication-server \
  --create-public-ip false

C) Replication server too small - increase the class and staging disk type:

# Change the default replication-server instance type and staging disks
# in the template (applies to newly created replication servers):
aws mgn update-replication-configuration-template \
  --replication-configuration-template-id rct-1234567890abcdef0 \
  --replication-server-instance-type t3.large \
  --default-large-staging-disk-type GP3 \
  --use-dedicated-replication-server

# For an existing source server, force the replication server to be
# recreated after changing settings:
aws mgn update-replication-configuration \
  --source-server-id s-1234567890abcdef0 \
  --replication-server-instance-type t3.large \
  --default-large-staging-disk-type GP3

D) Agent disconnects - unblock connectivity and reinstall the agent:

# On the source machine, check connectivity to the MGN endpoints (Linux):
curl -v https://mgn.eu-west-1.amazonaws.com 2>&1 | head -20   # port 443 - API
nc -vz <replication_server_IP> 1500                           # port 1500 - data

# Required outbound rules from the source machine:
#   443/TCP  -> mgn.<region>.amazonaws.com (registration and control)
#   1500/TCP -> replication server in the staging subnet (block transfer)

# If the agent is not reporting - reinstall it with fresh credentials:
sudo python3 aws-replication-installer-init.py \
  --region eu-west-1 \
  --aws-access-key-id AKIA... \
  --aws-secret-access-key ... \
  --no-prompt
# Use a dedicated IAM user with the AWSApplicationMigrationAgentPolicy,
# not administrative keys.

E) Repeated RESCAN - stabilise the source disks:

# For the duration of the migration, pause operations that generate large block changes on the VM:
#   - looping VMware snapshots / backups with change block tracking
#   - defragmentation, disk resizes, datastore moves
# If the rescan is caused by a real disk resize on the source,
# MGN has to re-scan it from scratch - schedule that outside the cutover window.

Prevention: start replication well ahead of time (days, not hours before cutover) and monitor lagDuration from day one. For large volumes, use a private network path and throttling matched to the link. Run a test-launch a few days before the actual cutover - boot errors (drivers, bootloader) are easier to fix without the pressure of the window.

Validation

# 1. All source servers in CONTINUOUS, lag near zero
aws mgn describe-source-servers \
  --query 'items[].{host:sourceProperties.identificationHints.hostname, state:dataReplicationInfo.dataReplicationState, lag:dataReplicationInfo.lagDuration}' \
  --output table
# Expected: dataReplicationState = "CONTINUOUS", lag = "PT0S" or near zero

# 2. Test-launch an instance and verify boot without affecting the cutover
aws mgn start-test \
  --source-server-ids s-1234567890abcdef0
# Then in EC2 check the test instance status checks:
aws ec2 describe-instance-status \
  --instance-ids i-0test... \
  --query 'InstanceStatuses[0].{sys:SystemStatus.Status, inst:InstanceStatus.Status}'
# Expected: sys = "ok", inst = "ok"; the machine boots and is reachable

# 3. After a successful test, mark and clean up the test instance
aws mgn mark-as-archived --source-server-id s-1234567890abcdef0   # only after cutover

If all servers are in CONTINUOUS, lag is stably near zero, and the test instance boots and passes status checks, replication is ready for cutover. Keep replication running continuously until the scheduled switchover.

Stalled MGN replication means you don't have a clean point to switch over from - the cutover window won't close and the migration date slips. The more dangerous scenario is "replication Healthy, but the test-launch produces a machine that won't boot": the problem only surfaces at the real cutover, when the production machines are already shut down. That's why the test-launch and boot verification are done a few days before the switchover, not during it.

 

Jerzy Kopaczewski

Is your VMware to AWS migration dragging on?

Book a free 30-minute call. We'll review your MGN configuration, replication network and sizing, and get the migration to a clean cutover.

Book a call