Setting the ESXi host persistent scratch partition to shared storage reverts to the default local OSDATA datastore after reboot

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0

Issue/Introduction

By default, the ESXi scratch partition is configured to a local OSDATA datastore.  After changing the ESXi scratch partition to a persistent location on shared storage (see: Creating a persistent scratch location for ESXi 8.x/7.x/6.x), and rebooting the host, the ScratchConfig.ConfiguredScratchLocation reverts back to the default location instead of the desired shared datastore.

When ESXi tries to activate the new scratch partition during the boot process, the shared storage VMFS volume designated for scratch has not been fully mounted yet and unavailable and scratch reverts to the defauly location:

YYYY-MM-DDTHH:MM:SS nativeExecutor[2098623]: SetupLegacyScratch: /vmfs/volumes/#######-#########-####-############/.locker-example-host: scratch path not active
YYYY-MM-DDTHH:MM:SS nativeExecutor[2098640]: Configured scratch location (/vmfs/volumes/#######-#########-####-############/.locker-example-host) is not available. Please verify that the path is accessible.
YYYY-MM-DDTHH:MM:SS Wa(12) Jumper2[2097749]: scratch has been redirected to /var/lib/vmware/osdata

........

YYYY-MM-DDTHH:MM:SS cpu76:2099912)LVM: 17386: File system '[Example-Datastore, #######-#########-####-############]' (LV #######-#########-####-############) mounted in 'rw' mode. 

The new scratch location in shared storage is not available during boot up, as the shared storage was not mounted yet . 

YYYY-MM-DDTHH:MM:SS cpu39:2098241)qedf:(2b:0.2):qedfc_link_update_handler:1846:Info: ST(LINK): LINK_UNKNOWN->LINK_UP
YYYY-MM-DDTHH:MM:SS cpu34:2098269)qedf:(2b:0.3):qedfc_link_update_handler:1846:Info: ST(LINK): LINK_UNKNOWN->LINK_UP
.........
YYYY-MM-DDTHH:MM:SS cpu24:2097842)ScsiClaimrule: 3470: Enabling claimrules for MP plugins.

Environment

VMware vSphere ESXi 7.0 U3
VMware vSphere ESXi 8.0 U3

Cause

When the host cannot find the configured path during boot, it automatically reverts to the default internal partition.  This behavior typically indicates a timing issue where the ESXi host finishes its initial boot sequence before the shared storage volume is fully "discovered" and mounted by the system.  A slow storage discovery process on the Cisco UCS fabric causes the VMFS volume to be unavailable when the ESXi configLocker script runs during boot. The jumpstart process is designed to continue as long as new paths or devices are being discovered. It will exit if no new device or path is detected within the last 3 seconds, which is controlled by the devListStabilityCount boot option.   

Resolution

Implement a longer wait time for storage-path-claim to discover devices by modifying the devListStabilityCount kernel boot parameter. By default it is 3 seconds. This could be increased to 10 seconds (or more) so that the volume/device with scratch gets discovered.

#> esxcli system settings kernel set -s devListStabilityCount -v 10

Reboot the ESXi host to apply the changes and verify the scratchconfig.configuredscratchlocation is set to the shared storage with the following command:

#> /bin/vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation