Trying to redeploy or update VSAN cluster nodes using Autodeploy in stateful mode fails
after the update attempt, the host boots up as stateless
vSphere Client shows a warning:
"Host does not appear to be booted into a stateful environment."
/var/run/log/syslog.log on the affected host show error entries like the ones below:
Syslog.log
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: ERROR: EngineModule::ApplyHostConfig. Exception: ("Errors: \nVolume '########-####-########d494' cannot be unmounted. Reason: Busy\n", b'')^@ [============== This points to OSDATA-XXXXXXXXXX and the Boot device]
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: EngineModule::ApplyHostConfig. Backtrace: ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/hostprofiles/tests/tools/hpcliModules/engineModule.py", line 549, in ApplyTaskList ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4722, in ApplyHostConfig ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4297, in ApplyGenericConfig ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/genericProfileBridge.py", line 928, in RecurseRemediateConfig ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/usr/lib/hostprofiles/plugins/systemCache/caching.py", line 1327, in RemediateConfig performGenericCacheTask(hostServices, task, diskName, overwriteVmfs) ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/usr/lib/hostprofiles/plugins/systemCache/caching.py", line 404, in performGenericCacheTask thin_partitions.installAction(persistUnmount=True) ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/usr/lib/vmware/weasel/thin_partitions.py", line 497, in installAction return systemStorageAction() ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/usr/lib/vmware/weasel/thin_partitions.py", line 754, in systemStorageAction autopartitionBootDisk(disk, keepDatastore, createDatastore) ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/systemStorage/installUtils.py", line 102, in autopartitionBootDisk ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/systemStorage/esxfs.py", line 344, in umountFileSystems ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/systemStorage/esxfs.py", line 338, in umountVmfsFileSystems ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/systemStorage/vmfsl.py", line 220, in vmfsUnmount ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/esxLogging.py", line 55, in callFuncRedirStderr ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/esxLogging.py", line 50, in callFuncRedirStderr ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: File "/lib64/python3.8/site-packages/esxutils.py", line 220, in runCli ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: Adding missing key EsxcliError.summary^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: excObj is of unknown type: <class 'esxutils.EsxcliError'>^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: INFO: Cleaned up Host Configuration^@
vmkernel.log
2023-01-31T04:50:40.731Z cpu15:2108426)WARNING: VC: 2366: unmounting opened volume ('########-####-########d494' 'OSDATA-########-####-########d494') with ref 2 is not allowed.
This is caused by the VSAN EPD service which still holds a lock on a file on this volume.
Note: Please be aware that using Autodeploy to reinstall/update ESXi hosts which part of a VSAN cluster is currently not officially supported.
Currently there is no resolution.
VMware engineering is aware of this problem.
Please subscribe to this article to be informed when the status changes.
To work around this issue, before attempting to redeploy a stateful host using Autodeploy, format the partitions on the boot device (BOOTBANK1, BOOTBANK2 and OSDATA) using external tools.