Error: "Host does not appear to be booted into a stateful environment" when attempting to redeploy or update vSAN clusters with Auto Deploy

Products

• VMware vSphere ESXi

Issue/Introduction

• Trying to redeploy or update vSAN cluster nodes using Auto Deploy in stateful mode fails after the update attempt, the host boots up as stateless.
• vSphere Client shows a warning:
  
  "Host does not appear to be booted into a stateful environment."
  
  
• /var/run/log/syslog.log on the affected host show error entries similar to:
  
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: ERROR: EngineModule::ApplyHostConfig. Exception: ("Errors: \nVolume '####-####-####-48df3704d494' cannot be unmounted. Reason: Busy\n", b'')  [==============è This points to OSDATA-####### and the Boot device]
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: EngineModule::ApplyHostConfig. Backtrace: 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/tests/tools/hpcliModules/engineModule.py", line 549, in ApplyTaskList 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4722, in ApplyHostConfig 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4297, in ApplyGenericConfig 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/genericProfileBridge.py", line 928, in RecurseRemediateConfig 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/hostprofiles/plugins/systemCache/caching.py", line 1327, in RemediateConfig     performGenericCacheTask(hostServices, task, diskName, overwriteVmfs) 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/hostprofiles/plugins/systemCache/caching.py", line 404, in performGenericCacheTask     thin_partitions.installAction(persistUnmount=True) 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/vmware/weasel/thin_partitions.py", line 497, in installAction     return systemStorageAction() 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/vmware/weasel/thin_partitions.py", line 754, in systemStorageAction     autopartitionBootDisk(disk, keepDatastore, createDatastore) 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/installUtils.py", line 102, in autopartitionBootDisk 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/esxfs.py", line 344, in umountFileSystems 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/esxfs.py", line 338, in umountVmfsFileSystems 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/vmfsl.py", line 220, in vmfsUnmount 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/esxLogging.py", line 55, in callFuncRedirStderr 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/esxLogging.py", line 50, in callFuncRedirStderr 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/esxutils.py", line 220, in runCli 
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: Adding missing key EsxcliError.summary
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: excObj is of unknown type: <class 'esxutils.EsxcliError'>
  2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: INFO: Cleaned up Host Configuration
  
  
• According to the vmkernel log (ESXi - /var/run/log/vmkernel.log), the volume cannot be unmounted:
  
  2023-01-31T04:50:40.731Z cpu15:2108426)WARNING: VC: 2366: unmounting opened volume ('####-####-####-48df3704d494' 'OSDATA-####-####-####-48df3704d494') with ref 2 is not allowed.

Cause

This is caused by the vSAN EPD service, which still holds a lock on a file on this volume.

Resolution

Currently, there is no resolution. VMware Engineering is aware of this problem. Subscribe to this article to be made aware when a fix is made available.

Note: Using Auto Deploy to reinstall or update ESXi hosts which are part of a VSAN cluster is currently not officially supported.

Workaround

• Before attempting to redeploy a stateful host using Auto Deploy, reformat the three system storage partitions (BOOTBANK1, BOOTBANK2, and OSDATA) using external tools.