Redeployment of ESXi 7.0 in VSAN cluster using stateful autodeploy fails with "Errors: \nVolume 'xxx' cannot be unmounted. Reason: Busy\n"
search cancel

Redeployment of ESXi 7.0 in VSAN cluster using stateful autodeploy fails with "Errors: \nVolume 'xxx' cannot be unmounted. Reason: Busy\n"

book

Article ID: 326257

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • Trying to redeploy or update VSAN cluster nodes using Autodeploy in stateful mode fails

  • after the update attempt, the host boots up as stateless

  • vSphere Client shows a warning:

"Host does not appear to be booted into a stateful environment."

  • /var/run/log/syslog.log on the affected host show error entries like the ones below:

Syslog.log
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: ERROR: EngineModule::ApplyHostConfig. Exception: ("Errors: \nVolume '6384676f-e212e71f-fd8d-48df3704d494' cannot be unmounted. Reason: Busy\n", b'')^@  [============== This points to OSDATA-XXXXXXXXXX and the Boot device]
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: EngineModule::ApplyHostConfig. Backtrace: ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/tests/tools/hpcliModules/engineModule.py", line 549, in ApplyTaskList ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4722, in ApplyHostConfig ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4297, in ApplyGenericConfig ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/genericProfileBridge.py", line 928, in RecurseRemediateConfig ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/hostprofiles/plugins/systemCache/caching.py", line 1327, in RemediateConfig     performGenericCacheTask(hostServices, task, diskName, overwriteVmfs) ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/hostprofiles/plugins/systemCache/caching.py", line 404, in performGenericCacheTask     thin_partitions.installAction(persistUnmount=True) ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/vmware/weasel/thin_partitions.py", line 497, in installAction     return systemStorageAction() ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/vmware/weasel/thin_partitions.py", line 754, in systemStorageAction     autopartitionBootDisk(disk, keepDatastore, createDatastore) ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/installUtils.py", line 102, in autopartitionBootDisk ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/esxfs.py", line 344, in umountFileSystems ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/esxfs.py", line 338, in umountVmfsFileSystems ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/vmfsl.py", line 220, in vmfsUnmount ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/esxLogging.py", line 55, in callFuncRedirStderr ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/esxLogging.py", line 50, in callFuncRedirStderr ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/esxutils.py", line 220, in runCli ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: Adding missing key EsxcliError.summary^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: excObj is of unknown type: <class 'esxutils.EsxcliError'>^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: INFO: Cleaned up Host Configuration^@
  • according to the VMKernel log (/var/run/log/vmkernel.log) the volume cannot be unmounted:
vmkernel.log
2023-01-31T04:50:40.731Z cpu15:2108426)WARNING: VC: 2366: unmounting opened volume ('6384676f-e212e71f-fd8d-48df3704d494' 'OSDATA-6384676f-e212e71f-fd8d-48df3704d494') with ref 2 is not allowed.
 


Environment

VMware vSphere ESXi 7.0.x
VMware vCenter Server 7.0.x

Cause

This is caused by the VSAN EPD service which still holds a lock on a file on this volume.

Note: Please be aware that using Autodeploy to reinstall/update ESXi hosts which part of a VSAN cluster is currently not officially supported.

Resolution

Currently there is no resolution.
VMware engineering is aware of this problem.
Please subscribe to this article to be informed when the status changes.


Workaround:

To work around this issue, before attempting to redeploy a stateful host using Autodeploy, format the partitions on the boot device (BOOTBANK1, BOOTBANK2 and OSDATA) using external tools.


Powered by Wolken Software