VMFS Datastore not mounting with error "No FS driver claimed device"
search cancel

VMFS Datastore not mounting with error "No FS driver claimed device"

book

Article ID: 451368

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • The datastore is not mounted to the ESXi host(s), although the backing device is seen by the host(s).
  • The datastore becomes inaccessible from the host(s).
  • VMs running on the specific datastore become invalid, or VM name is replaced with path to .vmx file in UI display.
  • In the var/run/log/vmkernel.log, the error message "No FS driver claimed device 'naa.#########:1': No filesystem on the device" may be seen.
  • In the var/run/log/vmkernel.log, the error message "("<Datastore name>") may be damaged on disk." may be seen.

Environment

VMware vSphere ESXi (All versions)

Cause

  • This occurs if LUNs are inadvertently presented to and formatted by another operating system (e.g., during a server recommissioning process). This can overwrite the GPT partition table and overwrite critical elements of the VMFS, effectively making the datastore unreadable to the ESXi host.
  • The below example hexdump indicates VMFS metadata in which the LVM header (beginning at offset 0020000) is overwritten with a repeating pattern.

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 1d 9a 00 00  |................|
000001c0  01 00 ee fe ff ff 01 00  00 00 ff ff ff ff 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  13 b2 c3 ec 00 00 00 00  01 00 00 00 00 00 00 00  |................|
00000220  ff 3f a9 d1 01 00 00 00  22 00 00 00 00 00 00 00  |.?......".......|
00000230  de 3f a9 d1 01 00 00 00  37 08 96 f6 30 f1 f9 43  |.?......#..0..#|
00000240  ae fd 89 f1 53 88 7a 18  02 00 00 00 00 00 00 00  |....#.z.........|
00000250  80 00 00 00 80 00 00 00  db 5d 4c 3c 00 00 00 00  |.........]##....|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  2a e0 31 aa 0f 40 db 11  95 90 00 0c 29 11 d1 b8  |*.1..@......)...|
00000410  c1 27 c6 60 c7 31 2c 4a  a8 ea 42 fc a7 95 a0 44  |.'.`.###..B....#|
00000420  00 08 00 00 00 00 00 00  de 3f a9 d1 01 00 00 00  |.........?......|
00000430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00200000  01 01 01 01 01 01 01 01  01 01 01 01 01 01 01 01  |................|     <<<<<<<
*
00200200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00201000  00 00 00 50 a3 03 00 00  02 00 00 00 00 00 00 00  |...P............|
00201010  01 00 00 00 35 63 61 65  32 32 64 32 2d 35 38 32  |....########-###|
00201020  31 34 34 64 38 2d 38 35  66 34 2d 38 30 33 30 65  |#####-####-#####|
00201030  30 30 34 62 30 64 34 00  00 00 00 00 00 00 00 00  |#######.........|
00201040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00201050  00 00 00 00 d2 22 ae 5c  d8 44 21 58 f4 85 80 30  |.....".\.###...0|
00201060  e0 04 b0 d4 01 00 00 00  b5 c7 d3 1a 30 86 05 00  |............0...|
00201070  00 00 00 00 35 3a 00 00  00 00 00 00 00 00 00 00  |....#:..........|
00201080  34 3a 00 00 00 00 00 00  12 cb d3 1a 30 86 05 00  |#:..........0...|

 

    • Here is an example of a healthy/expected LVM header pattern:

00200000  0d d0 11 c2 03 00 00 00  05 00 00 00 00 00 10 00  |................|     <<<<<<<   
00200010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00200020  44 61 74 61 73 74 6f 72  65 5f 50 72 6f 64 5f 30  |Datastore_Name_0|
00200030  31 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |1...............|
00200040  63 ad e2 5f 82 e1 c0 34  28 cb 00 50 56 a1 b2 c3  |c.._...4(..PV...|
00200050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00200060  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

 

  • When that happens, VOMA check shows "LVM magic not found at expected Offset,", as in the example:

[root@v1-vsphere-s2:~] voma -m vmfs -f check -d /vmfs/devices/disks/naa.#################:1
Running VMFS Checker version 2.1 in check mode
Initializing LVM metadata, Basic Checks will be done
Initializing LVM metadata..-
LVM magic not found at expected Offset,
It might take long time to search in rest of the disk.
Do you want to continue (Y/N)?

0) _Yes
1) _No

Select a number from 0-1:

Resolution

  1. Raise a Support Request with VMware to verify these findings. For more information, see Creating and managing Broadcom support cases 
  2. Once verified, restore from backups.
    In scenarios where the LVM metadata header has been overwritten or zeroed out, the VOMA repair tool will not run. The most reliable path to recovery is restoring virtual machines from valid, pre-corruption backups.
  3. If no backups available, consult with a data recovery organization as VMware does not perform data recovery. For more information, see Data recovery services for data not recoverable by VMware Technical Support.
Powered by Wolken Software