ESXi host upgrade fails due to error "[Errno 22] Invalid argument: '/scratch/lifecycle'"

Products

• VMware vSphere ESXi

Issue/Introduction

• Upgrading ESXi host through the vCenter fails 
• Upgrading through CLI shows the below error on the terminal :
  

      [OSError]
      [Errno 22] Invalid argument: '/scratch/lifecycle'
      Please refer to the log file for more details.

• Collecting logs from the host provides an empty /var/run/log directory.
• The /var/log/boot  file may have the below entries : 
  

  YYYY-MM-DDT02:04:26.006Z cpu13:2097682)WARNING: FS3: 608: VMFS volume storage/58######-##43####-a###-##9######ed8 on naa.60######00####f3################f:8 has been detected corrupted
  YYYY-MM-DDT02:04:25.396Z cpu12:2098415)WARNING: FS3: 608: VMFS volume OSDATA-64######-##24####-##3#-#c#########/64######-##24####-##3#-#c######### on naa.60######00####f3################f:7 has been detected corrupted

Environment

VMware vSphere ESXi 

Cause

A problematic scratch directory causes the '/scratch/lifecycle' directory to be inaccessible on the ESXi hosts causing the error during a host upgrade task. 

Resolution

Confirm if the scratch directory is damaged : 

• Check the current scratch directory using the below command : 
  

  vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation

• Verify if the directory is accessible from the value section of the above output. 
• Check if the directory is part of the VMFS volume that has been corrupted.

You may workaround this error by changing the scratch directory to a healthy VMFS volume using the below article 
Creating a persistent scratch location for ESXi 8.x/7.x