Local datastores disappear after re-Installing ESXi host using "Install ESXi, preserve VMFS datastore" option

Products

• VMware vSphere ESX 7.x

Issue/Introduction

If you are unable to roll back ESXi to a previous version using the Shift+R option, you may need to use the "Install ESXi, preserve VMFS datastore" option to reinstall without overwriting the VMFS datastore and restore the ESXi host configuration.

In a Nutanix environment, this procedure may result in the disappearance of local datastores.  As a result, the CVM virtual machine becomes unavailable.

Environment

VMware ESXi 7.0.X

Cause

For "Install ESXi, preserve VMFS datastore" option, local datastore preserves only the first VMFS partition.

Nutanix environments might have the following configuration:
 
 <Before re-Installing ESXi host uisng "Install ESXi, preserve VMFS datastore" option>

 esxcli storage core device partition list
 Device                                Partition  Start Sector  End Sector  Type  Size
 ------------------------------------  ---------  ------------  ----------  ----  ----
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          0                  0   468791999     0  240021504000
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          1                 64      204863     0     104857600
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          5             208896     8595455     6    4293918720
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          6            8597504    16984063     6    4293918720
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          7           16986112   268435455    f8  128742064128
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          8          268437504   426848926    fb   81106648576  <--datastore1 which is a VMFS snapshot
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          9          426848927   468789952    fb   21473805312  <--NTNX-local-ds-XXXX datastore
 
The default datastore, datastore1 on partition 8 is a VMFS snapshot and is not normally seen as a datastore, so it is not used. The NTNX-local-ds-XXXX datastore on partition 9 is used as a datastore for storing the CVM.

If you install on a local disk with this configuration using the "Install ESXi, preserve VMFS datastore" option, only the first partition 8 will be preserved, and partition 9 will not be preserved.

Note that datastore1 on the preserved partition 8 is a VMFS snapshot, so it will not be mounted automatically, and at first glance it will appear that no datastores have been preserved.
 
 <After re-Installing ESXi host uisng "Install ESXi, preserve VMFS datastore" option>

 esxcli storage core device partition list
 Device                                Partition  Start Sector  End Sector  Type  Size
 ------------------------------------  ---------  ------------  ----------  ----  ----
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          0                  0   468791999     0  240021504000
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          1                 64      204863     0     104857600
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          5             208896     8595455     6    4293918720
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          6            8597504    16984063     6    4293918720
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          7           16986112   268435455    f8  128742064128
 naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX          8          268437504   426848926    fb   81106648576  <--datastore1 which is a VMFS snapshot

Resolution

In order to resolve this issue, you must recreate partition 9.

IMPORTANT:  You must know the geometry of the partition before running the below commands.  Note that the partedUtil add command requires the following items (available from the output of esxcli storage core device partition list shown above):

Partition Number: 9
Start Sector:  426848927
End Sector: 468789952
GUID: AA31E02A400F11DB9590000C2911D1B8

Note:  The GUID will not change, that is the GUID for VMFS datastores.  For further information on the GUID, please see: Using partedUtil command line disk partitioning utility on ESXi

Example:

1. Add partition 9 (this is an example, use the correct partition number and disk geometry based on the output from your own host):

    partedUtil add /dev/disks/naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX gpt "9 426848927 468789952 AA31E02A400F11DB9590000C2911D1B8 0"

2. Run fixGpt :

    partedUtil fixGpt /dev/disks/naa.XXXXXXXXXXXXXXXXXXXXXXXXXXX

3. Reboot the ESXi host

After rebooting, the NTNX-local-ds-XXXX datastore will be visible and the CVM virtual machine will be available.

Additional Information

Manually creating a VMFS volume using vmkfstools -C:

https://knowledge.broadcom.com/external/article/309687/manually-creating-a-vmfs-volume-using-vm.html

Using partedUtil:

https://knowledge.broadcom.com/external/article/323144/using-partedutil-command-line-disk-parti.html