Capacity disks in a vSAN Original Storage Architecture (OSA) environment transition to an Unhealthy, Evacuated, or Unmounted state. This typically occurs during periods of high I/O, such as resynchronization or rebalancing operations.
Article ID: 443689
Updated On:
Products
Issue/Introduction
Symptoms:
- vSAN Physical Disk alarm:
Operation Healthis triggered. - Disk status shows as Evacuated or Unhealthy in Disk Management.
- Virtual machines experience performance degradation or brief stuns.
vobd.logcontains:[vob.vsan.lsom.devicewithhighlatency] vSAN device naa.#### has high latency. It will be evacuated and unmounted, consider replacing it.vsandevicemonitord.logcontains:WARNING - WRITE Average Latency on VSAN device naa.#### has exceeded threshold value 200000 us
Environment
- VMware vSAN 7.x
- VMware vSAN 8.x (OSA)
Cause
This issue is caused by physical hardware degradation of the storage media. vSAN Dying Disk Handling (DDH) detects sustained write latency exceeding thresholds (typically 200ms to 2500ms depending on configuration). vSAN proactively unmounts the device to prevent I/O congestion from impacting the rest of the cluster.
When a disk fails to complete I/O operations within this window multiple times, vSAN's Degraded Disk Handling (DDH) mechanism triggers a proactive evacuation of data to other healthy disks in the cluster and unmounts the degraded device to prevent it from causing further I/O congestion.
- In
var/run/log/vobd.log, the following entries are seen:
[vob.vsan.lsom.devicewithhighlatency] vSAN device naa.################ has high latency. It will be evacuated and unmounted, consider replacing it.- In
var/run/log/vsandevicemonitord.log, you will find specific latency triggers:
WARNING - WRITE Average Latency on VSAN device naa.################ has exceeded threshold value 2500000 us 1 times. Number of monitored intervals with excessive latency is 2 for device naa.################.- In
var/run/log/vmkernel.log, device naa.################ reports latency events:
Device naa.################ performance has improved. I/O latency reduced from ###### microseconds to ###### microseconds.Resolution
The degraded physical disks must be replaced to restore cluster health and redundancy.
Step 1: Identify the Physical Slot
- Identify the physical slot by lighting the locator LED using the following command.
esxcli storage core device set -d <naa.ID> --led-state locator --led-duration 100
Step 2: Logically Remove the Disk from vSAN
- Navigate to vSAN Cluster > Configure > vSAN > Disk Management.
- Select the host and the affected Disk Group.
- Select the degraded disk and click Remove Disk.
- Select No Data Migration when prompted.
- Note: Choosing data migration on a high-latency disk may cause the task to hang.
Step 3: Replace Hardware and Re-claim
- Physically replace the drive.
- Use Add Disks in the Disk Management UI to claim the new drive.
Additional Information
Feedback
