Physical drive failure on a storage array

Products

• VMware vSphere ESXi

Issue/Introduction

• Several virtual machines fail to power on
• You cannot restart the virtual machines
• A failed drive is detected on the storage array
• The RAID set experiences a fault
• In the /var/log/vmkernel.log file, you see entries similar to:
  
  Res3: 597: Inconsistency detected R/C 0, C/G 0, CG offset 0, CG size 0, totalR 0 #CGs 0
  FSS: 343: Failed with status 0xbad0012 for f530 28 1 xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx 0 0 0 0 0 0 0
  WARNING: Fil3: 1564: Failed to reserve volume f530 28 1 xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx 0 0 0 0 0 0 0
  
  
• The vmkernel.log file, you see entries similar to:
  
  WARNING: Fil3: 3304: Unknown object type 0
  WARNING: Fil3: 3304: Unknown object type 19267
  WARNING: FS3: 1575: Lock corruption detected at offset 0xc10000
  
  Or
  
  vmkernel: 0:04:57:24.056 cpu2:1039)WARNING: Fil3: 1791: Failed to reserve volume f530 28 1 xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx 0 0 0 0 0 0 0
  vmkernel: 0:04:57:24.056 cpu2:1039)FSS: 390: Failed with status Invalid metadata for f530 28 2 xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx 4 1 0 0 0 0 0
  vmkernel: [type 1] Invalid totalResources 0 (cluster 0).[type 1] Invalid nextFreeIdx 0 (cluster 0).0:02:33:31.812 cpu1:1041)WARNING: Res3: 1930: FS xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx may be damaged. Resource cluster metadata corruption detected
  
  vmkernel: 0:00:14:52.908 cpu4:1040)Res3: 644: Inconsistency detected R/C 0, C/G 0, CG offset 0, CG size 0, totalR 0 #CGs 0
  vmkernel: 0:00:14:52.908 cpu4:1040)SCSI: 5498: Logical device vml. xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx for target vmhba32:6:0 was registered successfully.
  
  Note: If you see these errors in the vmkernel.log file, check your storage array controller logs for reported drive failure.

Environment

VMware vSphere ESXi 7.X
VMware vSphere ESXi 8.X

Resolution

These symptoms are indicative of underlying storage failure (such as drive failure).

To resolve this issue, you must migrate the virtual machines from the affected storage and take corrective action. You may, for example, require a firmware update. For more information, contact your hardware vendor.