vSAN Disk Group Failure: High Congestion and Evacuation Timeouts from NVMe Cache Drive Wear-Out
search cancel

vSAN Disk Group Failure: High Congestion and Evacuation Timeouts from NVMe Cache Drive Wear-Out

book

Article ID: 447568

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

When a vSAN cache devices wears out, but does not fully fail,  in a VMware vSAN environment, a disk group may suddenly report an unhealthy state or drop offline completely, even if no raw capacity disks or NVMe storage components show immediate terminal failure in the UI. 

During this fault window, management operations via the vCenter Server Client may experience intermittent timeouts, and the affected ESXi host may fail to enter Maintenance Mode (MM) using the "Ensure Accessibility" data-safety spec.

Virtual machines may hang or crash.

Review of the vmkernel, vmkwarning, and vobd logs will reveal the following symptoms:

  • High volume of IORETRY throttling and unrecoverable read errors (UREs) mapping back to the native NVMe driver layer (status 0x281 / device internal error):
    WARNING: NVMEIO:2645 command 0x45df66dfabc0 failed: ctlr 257, queue 1, psaCmd 0x45df5a7fdbc0, status 0x281, opc 0x2, cid 799, nsid 1 WARNING: LSOMCommon: IORETRYParentIODoneCB:2536: Throttled: split status Read error
  • Continuous Cluster Level Object Manager (clomd) decommissioning timeouts (Failure Reason 12) stating that component data evacuation cannot progress:
    clomd[2099488] ####-HOST35: Failed to process entity decom state Timeout
    clomd[2099488] CLOMDecomFailDecommissioning: Going to fail decom with failure reason 12

Downstream capacity drives flooding retry logs and exhibiting latency spikes due to backend starvation, while vSAN Log Congestion values flood past the operational threshold of 200.

Environment

vSAN 8.x

All flash vSAN OSA with NVMe cache drives

Cause

The issue is caused by the severe physical media degradation of the primary NVMe tier caching drive (e.g., running beyond ~49,000 continuous power-on hours). Once the drive exhausts its on-board over-provisioned blocks, it begins failing block-level read operations with internal hardware exceptions (status 0x281).

The specific driver failures are traced via the following log signatures:

####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu107:2098628)WARNING: NVMEIO:2645 command 0x45df66dfabc0 failed: ctlr 257, queue 1, psaCmd 0x45df5a7fdbc0, status 0x281, opc 0x2, cid 799, nsid 1
####-##-##T##:##:##.###Z In(182) vmkernel: cpu92:2097374)LSOMCommon: IORETRYCompleteIO:469: Throttled: 0x45dd2c709200 IO type 264 (READ) isOrdered:NO isSplit:YES isEncr:NO since 1206 msec status Read error
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu92:2097374)WARNING: PLOG: PLOG_MarkCacheError:1302: Disk: naa.################ has encountered error status: Read error. Marking the disk for unmount-mount operation by the device monitor daemon

This structural failure is corroborated by the physical device metrics, which explicitly show worn media reallocation depletion:

SMART Data for Disk : t10.################1
Parameter Value Threshold Worst Raw  
-----------------------------------------------------------  
Health Status OK N/A N/A N/A  
Power-on Hours 49677 N/A N/A N/A  
Power Cycle Count 64 N/A N/A N/A  
Reallocated Sector Count 2 90 N/A N/A  
Drive Temperature 27 70 N/A N/A

If any vSAN objects are inaccessible and the vSAN Dying Disk Handling (DDH) process, or a manual Maintenance Mode task, attempts to programmatically unmount the disk group, the evacuation engine encounters an Ensure Data Availability policy constraint. Because the degraded flash media cannot process outbound read commands to copy out components belonging to inaccessible objects, the decommissioning engine eventually times out as it cannot safely remove the disk group. The drive remains stuck in the active storage datapath, forcing the cache tier to fill leading to vSAN congestion, backing up local I/O queues, and causing wide-scale management latency as hostd is impacted.

A standard warm reboot does not clear this condition as standby power remains active on the PCIe lanes, and the NVMe device remains in a soft failure condition.

Resolution

To resolve this issue and safely replace the degraded hardware, follow the steps below:

Step 1: Clear the Maintenance Mode Blocker

  1. Log into the vSphere Client and navigate to the vSAN Virtual Health dashboard.

  2. Review the vSAN Skyline health check or the Virtual Objects tab to locate the specific objects currently flagged as Inaccessible.

  3. Identify the inaccessible objects.

  4. Manually delete these objects with the help of Support.

Step 2: Enter Maintenance Mode

  1. Once the inaccessible objects are deleted, restart Maintenance Mode entering using the Ensure Accessibility option. The programmatic engine will now immediately clear the unmount workflow.

  2. If the NVMe storage controller remains completely locked up or non-responsive under persistent log congestion, execute a full Cold Boot (hard power cycle). Completely draining the capacitors on the motherboard's PCIe slots will reset the drive's microcode and temporarily restore enough basic sector responsiveness to complete the Maintenance Mode process.

  3. Once the host is in Maintenance Mode or booting the impact to running VMs will subside.

Step 3: Replace the Cache device and Recreate Disk Group

  1. With the ESXi host safely stabilized inside Maintenance Mode, navigate to Cluster > Configure > vSAN > Disk Management.

  2. Select the affected host node, locate the degraded disk group stub, and click Remove. If this process fails open a case with Support to assist.

  3. Power down the physical server chassis and physically replace the cache NVMe drive according to your hardware vendor's instructions.

  4. Power on the host, and recreate the disk group in the UI.

  5. Allow any vSAN resync to complete, and validate vSAN Skyline health shows no related errors.
Powered by Wolken Software