Upgrading host from 8.0 U2 using stateful auto deploy boot fails to complete with error "cannot be unmounted. Reason: Busy"

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• Issue is seen post upgrading the host to ESXi 8.0 Update 2
• in /var/run/log/syslog.log the following entries can be found:

  <timestamp> Wa(12) hostprofile[2099609]: opID=MainThread: Core dump file is enabled. Disabling it
  <timestamp> Wa(12) hostprofile[2099609]: opID=MainThread: Shutting off coredump file: 0:True
  <timestamp> In(14) hostprofile[2099609]: opID=MainThread: Now caching to disk: t10.ATA_____#########################________________________###########
  <timestamp> Er(11) hostprofile[2099609]: opID=MainThread: EngineModule::ApplyHostConfig. Exception: ("Errors: \nVolume '########-########-####-############' cannot be unmounted. Reason: Busy\n", b'')
  <timestamp> Wa(12) hostprofile[2099609]: opID=MainThread: EngineModule::ApplyHostConfig. Backtrace:
  <timestamp> Wa(12)[+] hostprofile[2099609]:
  <timestamp> Wa(12) hostprofile[2099609]: opID=MainThread:    File "/lib64/python3.8/site-packages/hostprofiles/tests/tools/hpcliModules/engineModule.py", line 553, in ApplyTaskList
  <timestamp> Wa(12)[+] hostprofile[2099609]:
  <timestamp> Wa(12) hostprofile[2099609]: opID=MainThread:    File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4724, in ApplyHostConfig
  <timestamp> Wa(12) hostprofile[2099609]: opID=MainThread: Adding missing key EsxcliError.summary
  <timestamp> Wa(12) hostprofile[2099609]: opID=MainThread: excObj is of unknown type: <class 'esxutils.EsxcliError'>
  <timestamp> In(14) hostprofile[2099609]: opID=MainThread: Cleaned up Host Configuration
  <timestamp> In(14) hostprofile[2099609]: opID=MainThread: /sbin/applyHostProfile sending boot msg to console: Applying Host Profile task list...Error
  <timestamp> Er(11) hostprofile[2099609]: opID=MainThread: while applying tasks: (vmodl.fault.SystemError) {

 

Environment

VMware vSphere ESXi 8.0.2

Cause

When the chosen system cache partition is in use, it cannot be reformatted/unmounted. The stateful install or system cache fails due to one of the following causes:

1. The partition is used by OSData 
2. A log file is hardcoded to some file path of that partition

Resolution

To resolve the issue, please patch the ESXi hosts to vSphere ESXi 8.0 Update 3e Build 24674464.

In order to workaround the issue, implement the steps below : 

• Log in to the vCenter server using UI
• Navigate to the desired cluster
• Click Configure on the right hand pane -> select vSphere Availability -> Edit
• Go in Edit Cluster Settings and click Advanced Options
• Add the following Option:

Option: das.config.fdm.dumpDirectory
Value:   /var/log

• Click Ok
• Perform a reconfigure HA on all the hosts in the cluster

 

Additional Information

Note: Do not disable vSphere HA if there are FT (Fault Tolerance) VM running on the cluster