Upgrading host from vSphere ESXi 6.x to 7.x datastores may become overwritten if incorrect scratch location is set, prior, or after upgrade

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• After upgrading a host from vSphere ESXi 6.x to 7.x, you may see the datastore along with the vms/disks located inside become inaccessible. In the vmkernel.log, you will see error message similar to:

xxxx-xx-xxT23:20:39.931Z cpu51:2100016)LVM: 7013: Failed to probe the VMFS header of naa.6742b0f00000053600000000000ef624:7 due to No filesystem on the device
xxxx-xx-xxT23:20:39.942Z cpu51:2100016)LVM: 7013: Failed to probe the VMFS header of naa.6742b0f00000053600000000000ef624:7 due to No filesystem on the device
xxxx-xx-xT23:20:39.944Z cpu51:2100016)LVM: 7013: Failed to probe the VMFS header of naa.6742b0f00000053600000000000ef624:7 due to No filesystem on the device

• In the datastore you will see vmfs-l label instead of vmfs.

Storage device from 7.0 host:

partedUtil getptbl /vmfs/devices/disks/naa.6742b0f00000053600000000000ef6e4
gpt
1069397 255 63 17179869184
7 2048 268435455 4EB2EA3978554790A79EFAE495E21F8D vmfsl 0

Note: If you see the same from a 6.7 host, the vmfsl label may show as unknown in 6.7, as 6.7 does not recognize the vmfsl label.

Note: This can also happen if you upgrade to 7.x and change the scratch location to the root of a VMFS datastore and then reboot. 

Environment

VMware vSphere ESXi 7.0.x
VMware ESXi 6.7.x

Cause

This issue is caused when the scratch location is set up incorrectly to the root of a datastore.

Resolution

This issue is resolved in VMware ESXi 7.0  P03 GA

Workaround:
If you encountered this issue, unfortunately, there is no option from a VMWare perspective to recover any lost data on this volume. You will need to restore your data from backups.

To avoid encountering this issue, ensure all the systems have scratch configured to a unique subdirectory if you decide to use a datastore file location, see Creating a persistent scratch location for ESXi 7.x/6.x/5.x/4. This policy should be checked prior to upgrading and should continue beyond upgrading as this is best practices regardless of this issue. The directory is recommended to have the ESXi host name or other host identifier so it is unique to that host and identifiable. 

To set scratch:

1. Click on Advanced System settings of host in the vCenter Server.
2. Click on Edit.
3. Change the ScratchConfig.ConfiguredScratchLocation configuration option and set it to the full path of the directory including the unique sub folder:

For Example: 
/vmfs/volumes/DatastoreName/.locker-ESXi-hostname

Note: Reboot is required to apply the change.

Additional Information

To set scratch location, see Creating a persistent scratch location for ESXi 7.x/6.x/5.x/4.x