In a VMware vSAN OSA enabled environments, one or more capacity disks are marked as Unhealthy and automatically Evacuated or Unmounted.
Article ID: 443689
Updated On:
Products
Issue/Introduction
Users may observe the following symptoms in the vSphere Client:
- vSAN Physical Disk alarm: 'Operation Health'.
- Disk status shows as Evacuated or Unhealthy in Disk Management.
- Virtual Machine experience performance degradation or brief 'stuns'.
Environment
vSAN 7.x
vSAN 8.x (OSA)
Cause
This issue is caused by physical hardware degradation of the storage media. The vsandevicemonitord service/Degraded Disk Handling detects sustained write latency exceeding the default threshold (typically 2,500ms).
When a disk fails to complete I/O operations within this window multiple times, vSAN's Degraded Disk Handling (DDH) mechanism triggers a proactive evacuation of data to other healthy disks in the cluster and unmounts the degraded device to prevent it from causing further I/O congestion.
In var/run/log/vobd.log, the following entries are seen:
[vob.vsan.lsom.devicewithhighlatency] vSAN device naa.################ has high latency. It will be evacuated and unmounted, consider replacing it.
In var/run/log/vsandevicemonitord.log, you will find specific latency triggers:
WARNING - WRITE Average Latency on VSAN device naa.################ has exceeded threshold value 2500000 us 1 times. Number of monitored intervals with excessive latency is 2 for device naa.################.
In var/run/log/vmkernel.log, device naa.################ is reporting latency events:
Device naa.################ performance has improved. I/O latency reduced from ###### microseconds to ###### microseconds.Resolution
The degraded physical disks must be replaced to restore cluster health and redundancy.
Step 1: Identify the Physical Slot
Use the ESXi command line to light the locator LED for the degraded disk:
esxcli storage core device set -d <naa.ID> --led-state locator --led-duration 100Step 2: Logically Remove the Disk from vSAN
1. Navigate to the vSAN Cluster > Configure > vSAN > Disk Management.
2. Select the host containing the failed disk.
3. Locate the affected Disk Group and select the degraded disk.
4. Click Remove Disk.
5. When prompted for Data Migration, select No Data Migration.
- Note: Since the disk is already evacuated or in a high-latency state, attempting a full migration now can cause tasks to hang or further impact performance.
Step 3: Replace Hardware and Re-claim
1. Physically replace the drive in the identified slot.
2. Return to the Disk Management UI and use the Add Disks option to claim the new drive into the existing disk group.
3. Monitor the vSAN Skyline Health to ensure the disk is healthy and resyncing tasks complete.
Additional Information
Feedback
