Unable to mount the datastore after unmounting.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• In vCenter, the datastore will be detected as unmounted from hosts.

• Run the command "esxcli storage filesystem list". Mount status will show as false.

  Mount Point                                        Volume Name                                 UUID                                 Mounted  Type    Size            Free
-------------------------------------------------  ------------------------------------------  -----------------------------------  -------  ------  --------------  ----
                                                   Volume name                                 ########-########-####-############   false    VMFS-6               0               0

• No errors observed while mounting from CLI, vCenter or Host UI

Validation:

• From the vmkernel logs, We also see the metadata inconsistency detected in the VMFS file system layer, specifically related to resource and cluster group calculations on the datastore.

YYYY-MM-DDThh:mm:ss.ms cpu46:59237035)Res3: 2517: Netapp01: Inconsistency detected for type 2 (totalR 120259108352, rPerC 256, cPerG 8, numCG 12)
YYYY-MM-DDThh:mm:ss.ms cpu46:59237035)Vol3: 4462: Failed to get object 28 type 1 uuid ########-########-####-############ FD 0 gen 0 :No filesystem on the device
YYYY-MM-DDThh:mm:ss.ms cpu46:59237035)WARNING: Fil3: 1539: Failed to reserve volume f532 28 1 ######## ######## #### ############ 0 0 0 0 0 0 0

• VOMA dump indicates that the values retrieved from the metadata are significantly different from what the VOMA tool expects.

[root@VM2:/vmfs/volumes/########-########-####-############] voma -f check -Z naa.################################_extracted.dmp
Module name is missing. Using "vmfs" as default
Running VMFS Checker version 2.1 in check mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
   Detected VMFS-6 file system (labeled:'Netapp01') with UUID:5ffd7811-13b74108-fadb-08f1eaec621a, Version 6:82
 ON-DISK ERROR: Unexpected 'parentResourcesPerCluster' 14852038, expected 0
 ON-DISK ERROR: Unexpected 'numClusterGroups' 12, expected 58720268
         ERROR: Failed to check sbc.sf.
   VOMA failed to check device : Out of memory

Total Errors Found: 2

• The object retrieval is failing because critical filesystem components are unreadable or missing.

• From the Hexdump file it is noticed the VMFS data is over written.

hexdump -C /vmfs/devices/disks/naa.################################ | less

018f8000  01 ef cd ab 00 80 6f 00  00 00 00 00 00 00 00 00  |......o.........|
018f8010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
018f9000  01 ef cd ab 00 90 6f 00  00 00 00 00 00 00 00 00  |......o.........|
018f9010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
018fa000  01 ef cd ab 00 a0 6f 00  00 00 00 00 00 00 00 00  |......o.........|
018fa010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Resolution

• In circumstances where the corruption is irreversible, VMware recommends that you restore from backup or consult with a data recovery organization as VMware does not perform data recovery. For more information, see Data recovery services for data not recoverable by VMware Technical Support.

Additional Information

Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency