Physical drive failure on a storage array

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machines fail to power on
• Cannot restart the virtual machines
• A failed drive is detected on the storage array
• The RAID set on the storage array experiences a fault
• In the /var/log/vmkernel.log file, there may be entries similar to:
  
  Res3: 597: Inconsistency detected R/C 0, C/G 0, CG offset 0, CG size 0, totalR 0 #CGs 0
  FSS: 343: Failed with status 0xbad0012 for f530 28 1 xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx 0 0 0 0 0 0 0
  WARNING: Fil3: 1564: Failed to reserve volume f530 28 1 xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx 0 0 0 0 0 0 0
  
  
• In the /var/log/vmkernel.log file, there may be entries similar to:
  
  WARNING: Fil3: 3304: Unknown object type 0
  WARNING: Fil3: 3304: Unknown object type 19267
  WARNING: FS3: 1575: Lock corruption detected at offset 0xc10000
  
  Or
  
  vmkernel: 0:04:57:24.056 cpu2:1039)WARNING: Fil3: 1791: Failed to reserve volume f530 28 1 xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx 0 0 0 0 0 0 0
  vmkernel: 0:04:57:24.056 cpu2:1039)FSS: 390: Failed with status Invalid metadata for f530 28 2 xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx 4 1 0 0 0 0 0
  vmkernel: [type 1] Invalid totalResources 0 (cluster 0).[type 1] Invalid nextFreeIdx 0 (cluster 0).0:02:33:31.812 cpu1:1041)WARNING: Res3: 1930: FS xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx may be damaged. Resource cluster metadata corruption detected
  
  vmkernel: 0:00:14:52.908 cpu4:1040)Res3: 644: Inconsistency detected R/C 0, C/G 0, CG offset 0, CG size 0, totalR 0 #CGs 0
  
  Note: If these errors are seen in the vmkernel.log file, check the storage array controller logs for a reported drive failure.

Environment

VMware vSphere ESXi 6.5.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x

Cause

These symptoms are indicative of underlying storage failure, such as a drive failure. 

Resolution

To resolve this issue, migrate the virtual machines from the affected storage.

Additional Information

For more information, contact the storage hardware vendor.