Logo
Hetzner dedicated-servers bare-metal EU-hosting RAID

Hetzner dedicated server rescue mode: rebuild mdadm RAID after disk replacement

Rebuild an mdadm software RAID array on a Hetzner dedicated server after disk replacement using rescue mode, partitioning, and array reassembly.

·
Hetzner replaced the failed disk in your dedicated server. The new disk is blank. Your server won't boot because the RAID is degraded. Boot into rescue mode, partition the replacement disk, add it to the array, and get back to normal.

For broader EU dedicated server management, see our dedicated server operations page. For EU data residency considerations when choosing a hosting provider, see our case study on moving infrastructure to the EU.

Symptoms

Your Hetzner dedicated server is unresponsive or booted into degraded mode. The Hetzner Robot panel shows a disk replacement was performed:

# Hetzner Robot notification:
# Subject: Hardware replacement completed for server EX44 (10.0.0.1)
# Body: Drive /dev/sdb has been replaced. Server is in rescue mode.

Or you notice the RAID is degraded after a reboot:

# From rescue mode or degraded boot
cat /proc/mdstat
# md0 : active raid1 sda1[0]
#       524224 blocks super 1.2 [2/1] [U_]
# md1 : active raid1 sda2[0]
#       1953382336 blocks super 1.2 [2/1] [U_]

The [U_] pattern means one disk is missing from the mirror.

Cause

Hetzner’s hardware team physically replaced the failed drive but the new disk arrives blank (no partition table, no mdadm superblock). The software RAID array (md0, md1, etc.) is running degraded on the remaining good disk. The server may:

  • Boot into rescue mode automatically (if Hetzner support triggered rescue)
  • Boot normally in degraded mode (if the good disk has the boot loader)
  • Fail to boot entirely (if the good disk isn’t configured as boot fallback)

You need to: partition the new disk identically to the surviving one, add each partition to the corresponding mdadm array, wait for rebuild, and ensure GRUB is installed on both disks.

Fix

Step 1: Boot into rescue mode via Hetzner Robot

If not already in rescue mode:

  1. Log into Hetzner Robot
  2. Select your server → Rescue tab
  3. Activate rescue system (Linux 64-bit)
  4. Note the temporary root password
  5. Trigger a hardware reset (Reset tab → “Send Ctrl+Alt+Del” or “Execute automatic hardware reset”)
  6. SSH into the rescue system: ssh root@your-server-ip

Step 2: Identify disks and existing array layout

# List all disks
lsblk -o NAME,SIZE,TYPE,FSTYPE,MODEL

# Check which disk is new (no partitions)
fdisk -l /dev/sda
fdisk -l /dev/sdb

# Check existing mdadm arrays (running on the good disk)
cat /proc/mdstat

# Get partition layout from the good disk
sgdisk -p /dev/sda
# Note: sda is assumed as the surviving disk. Adjust if sdb survived.

Step 3: Clone partition table to the new disk

# Copy partition table from good disk (sda) to new disk (sdb)
sgdisk -R /dev/sdb /dev/sda

# Randomize the GUIDs on the new disk (required for GPT)
sgdisk -G /dev/sdb

# Verify partition layout matches
sgdisk -p /dev/sdb

Step 4: Add partitions to mdadm arrays

# Add each partition to its corresponding array
# Adjust md device numbers and partition numbers for your setup

mdadm --add /dev/md0 /dev/sdb1
mdadm --add /dev/md1 /dev/sdb2

# If you have a swap partition on md:
# mdadm --add /dev/md2 /dev/sdb3

# Check rebuild started
cat /proc/mdstat
# Expected: [2/1] [U_] with recovery percentage showing

Step 5: Wait for rebuild to complete

# Monitor rebuild progress
watch -n 10 cat /proc/mdstat

# For a 2 TB disk, expect ~2-4 hours on SATA, ~30-60 min on NVMe
# You can speed it up:
echo 200000 > /proc/sys/dev/raid/speed_limit_min
echo 500000 > /proc/sys/dev/raid/speed_limit_max

Step 6: Install GRUB on both disks

# Mount the root filesystem
mount /dev/md1 /mnt

# Bind mount required filesystems
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys

# If separate /boot partition (md0):
mount /dev/md0 /mnt/boot

# Chroot and install GRUB on both disks
chroot /mnt
grub-install /dev/sda
grub-install /dev/sdb
update-grub
exit

# Unmount
umount /mnt/boot 2>/dev/null
umount /mnt/sys
umount /mnt/proc
umount /mnt/dev
umount /mnt

Step 7: Disable rescue mode and reboot

  1. In Hetzner Robot → Rescue tab → Deactivate rescue system
  2. Reboot: reboot
  3. Wait 2-3 minutes, then SSH into your server normally

Validation

# 1. Verify RAID is fully synced
cat /proc/mdstat
# Expected: [2/2] [UU] for all arrays (no rebuild in progress)

# 2. Verify both disks are present in each array
mdadm --detail /dev/md0 | grep -E "State|Devices|/dev/"
# Expected: State: clean, Active Devices: 2

mdadm --detail /dev/md1 | grep -E "State|Devices|/dev/"
# Expected: State: clean, Active Devices: 2

# 3. Verify GRUB is installed on both disks
dd if=/dev/sda bs=512 count=1 2>/dev/null | strings | grep -i grub
dd if=/dev/sdb bs=512 count=1 2>/dev/null | strings | grep -i grub
# Expected: GRUB string found on both

# 4. Test boot from secondary disk (optional but recommended)
# In Robot → Reset → Boot next from /dev/sdb (if available)

# 5. Verify filesystem health
dmesg | grep -i "error\|fail\|corrupt"
# Expected: no filesystem errors

If all arrays show [UU] (both disks active), GRUB is on both disks, and no filesystem errors appear in dmesg, the rebuild is complete. Your server is back to full redundancy.

Running a dedicated server on degraded RAID means one more disk failure equals total data loss. The window between Hetzner’s disk replacement and your RAID rebuild is the highest risk moment in the lifecycle of that server. The faster you rebuild, the shorter the exposure. For production servers, we recommend monitoring mdadm status with alerts and having this procedure documented and tested before you need it.

 

Jerzy Kopaczewski

Managing dedicated servers in EU datacenters?

Book a free 30-minute call. We help teams operate bare-metal infrastructure at Hetzner, OVH, and Scaleway with proper monitoring and disaster recovery.