Proxmox Ceph OSD marked out: why recovery is slow and how to speed it up
Speed up Ceph recovery in Proxmox when an OSD is marked out and PG backfill crawls despite available disk bandwidth.
out. Recovery started but it's crawling at 10-50 MB/s despite NVMe drives capable of gigabytes per second. Your cluster is running degraded longer than it should. Here's how to unblock it.
For broader private cloud architecture guidance, see our Proxmox private cloud consulting page. If you’re running Kubernetes on top of Proxmox, our Kubernetes on Proxmox case study covers the full storage layer design.
Symptoms
Ceph shows degraded PGs and recovery is active but extremely slow:
# Check cluster health
ceph health detail
# HEALTH_WARN 1 osds down; Degraded data redundancy: 847/25410 objects degraded
# Check recovery speed
ceph -s
# recovery: 12 MiB/s, 3 objects/s
# The OSD is marked out
ceph osd tree | grep -i down
# 4 hdd 3.63000 osd.4 down 0 1.00000
# PG status shows backfilling
ceph pg stat
# 256 pgs: 241 active+clean, 15 active+recovering+degraded
Recovery should complete in minutes on modern hardware but instead takes hours. The cluster stays in HEALTH_WARN state, which means any second disk failure during this window risks data loss.
Cause
Ceph deliberately throttles recovery traffic to avoid starving client I/O. The default settings are conservative:
osd_recovery_max_activedefaults to 3 (only 3 concurrent recovery operations per OSD)osd_recovery_sleepdefaults to 0 (but some distributions set 0.1s)osd_max_backfillsdefaults to 1 (only 1 backfill operation per OSD at a time)osd_recovery_op_priorityis lower than client op priority by default
These defaults are reasonable for HDD-based clusters under heavy client load. On NVMe or SSD clusters, or during maintenance windows with low client traffic, they’re unnecessarily conservative and extend the degraded window.
Additionally, Proxmox’s default Ceph deployment inherits these values without tuning for the actual hardware underneath.
Fix
Step 1: Check current recovery parameters
# Check active recovery settings
ceph config get osd osd_recovery_max_active
ceph config get osd osd_max_backfills
ceph config get osd osd_recovery_sleep
ceph config get osd osd_recovery_op_priority
Step 2: Temporarily increase recovery throughput
For NVMe/SSD clusters during a maintenance window or low-traffic period:
# Increase concurrent recovery operations
ceph config set osd osd_recovery_max_active 5
# Allow more backfills simultaneously
ceph config set osd osd_max_backfills 3
# Remove any artificial sleep between recovery ops
ceph config set osd osd_recovery_sleep 0
# Raise recovery priority (63 = equal to client ops)
ceph config set osd osd_recovery_op_priority 63
For HDD clusters (lower these values):
ceph config set osd osd_recovery_max_active 3
ceph config set osd osd_max_backfills 2
ceph config set osd osd_recovery_sleep 0.05
Step 3: Monitor recovery progress
# Watch recovery speed in real time
watch -n 5 'ceph -s | grep -E "recovery|degraded|misplaced"'
# Expected: recovery speed should jump significantly
# NVMe cluster: 200-500 MiB/s
# SSD cluster: 100-300 MiB/s
# HDD cluster: 50-150 MiB/s
Step 4: Restore defaults after recovery completes
Once the cluster returns to HEALTH_OK:
# Reset to safe defaults for normal operation
ceph config rm osd osd_recovery_max_active
ceph config rm osd osd_max_backfills
ceph config rm osd osd_recovery_sleep
ceph config rm osd osd_recovery_op_priority
Step 5: If the OSD should come back online (disk is healthy)
If the disk is fine and the OSD just crashed temporarily:
# Start the OSD (on the Proxmox node where it lives)
systemctl start ceph-osd@4
# Mark it back in (if not auto-marked)
ceph osd in osd.4
# Verify it rejoins
ceph osd tree | grep osd.4
Permanent tuning for NVMe clusters:
If your cluster is all-NVMe and you want faster recovery as the default:
# Set as persistent config (survives restarts)
ceph config set osd osd_recovery_max_active 5 --force
ceph config set osd osd_max_backfills 3 --force
Or in Proxmox GUI: Datacenter → Ceph → Configuration → OSD → set the parameters there.
Validation
# 1. Verify cluster health is restored
ceph health
# Expected: HEALTH_OK
# 2. Verify no degraded PGs remain
ceph pg stat
# Expected: all PGs "active+clean"
# 3. Verify all OSDs are up and in
ceph osd tree | grep -c "up"
# Expected: matches your total OSD count
# 4. Verify recovery settings are back to defaults
ceph config get osd osd_recovery_max_active
# Expected: 3 (default)
# 5. Check no slow requests during recovery
ceph daemon osd.0 perf dump | jq '.osd.op_latency.avgcount'
# Expected: no abnormal spike in client operation latency
If ceph health shows HEALTH_OK and all PGs are active+clean, the recovery is complete. The cluster is back to full redundancy.
A degraded Ceph cluster is one disk failure away from data loss. The default recovery throttling extends this risk window from minutes to hours on modern hardware. For production clusters, tuning recovery parameters to match your actual disk performance is the difference between a 15-minute HEALTH_WARN and an overnight window where a second failure means calling your backup provider.
Running Proxmox + Ceph in production?
Book a free 30-minute call. We design and operate Proxmox clusters with Ceph storage tuned for your hardware and workload profile.