Redeployment of ESXi 7.0 in VSAN cluster using stateful autodeploy fails with "Errors: \nVolume 'xxx' cannot be unmounted. Reason: Busy\n"
search cancel

Redeployment of ESXi 7.0 in VSAN cluster using stateful autodeploy fails with "Errors: \nVolume 'xxx' cannot be unmounted. Reason: Busy\n"

book

Article ID: 326257

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

  • Trying to redeploy or update VSAN cluster nodes using Autodeploy in stateful mode fails
  • after the update attempt, the host boots up as stateless

  • vSphere Client shows a warning:

"Host does not appear to be booted into a stateful environment."

  • /var/run/log/syslog.log on the affected host show error entries like the ones below:

Syslog.log
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: ERROR: EngineModule::ApplyHostConfig. Exception: ("Errors: \nVolume '########-####-########d494' cannot be unmounted. Reason: Busy\n", b'')^@  [============== This points to OSDATA-XXXXXXXXXX and the Boot device]
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: EngineModule::ApplyHostConfig. Backtrace: ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/tests/tools/hpcliModules/engineModule.py", line 549, in ApplyTaskList ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4722, in ApplyHostConfig ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4297, in ApplyGenericConfig ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/hostprofiles/pyEngine/genericProfileBridge.py", line 928, in RecurseRemediateConfig ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/hostprofiles/plugins/systemCache/caching.py", line 1327, in RemediateConfig     performGenericCacheTask(hostServices, task, diskName, overwriteVmfs) ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/hostprofiles/plugins/systemCache/caching.py", line 404, in performGenericCacheTask     thin_partitions.installAction(persistUnmount=True) ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/vmware/weasel/thin_partitions.py", line 497, in installAction     return systemStorageAction() ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/usr/lib/vmware/weasel/thin_partitions.py", line 754, in systemStorageAction     autopartitionBootDisk(disk, keepDatastore, createDatastore) ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/installUtils.py", line 102, in autopartitionBootDisk ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/esxfs.py", line 344, in umountFileSystems ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/esxfs.py", line 338, in umountVmfsFileSystems ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/systemStorage/vmfsl.py", line 220, in vmfsUnmount ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/esxLogging.py", line 55, in callFuncRedirStderr ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/esxLogging.py", line 50, in callFuncRedirStderr ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING:   File "/lib64/python3.8/site-packages/esxutils.py", line 220, in runCli ^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: Adding missing key EsxcliError.summary^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: WARNING: excObj is of unknown type: <class 'esxutils.EsxcliError'>^@
2023-02-08T04:59:36Z Host Profiles[2107775 opID=MainThread]: INFO: Cleaned up Host Configuration^@
  • according to the VMKernel log (/var/run/log/vmkernel.log) the volume cannot be unmounted:
vmkernel.log
2023-01-31T04:50:40.731Z cpu15:2108426)WARNING: VC: 2366: unmounting opened volume ('########-####-########d494' 'OSDATA-########-####-########d494') with ref 2 is not allowed.

Environment

  • VMware vSphere ESXi 7.0.x
  • VMware vCenter Server 7.0.x

Cause

This is caused by the VSAN EPD service which still holds a lock on a file on this volume.

Note: Be aware that using Autodeploy to reinstall/update ESXi hosts which part of a VSAN cluster is currently not officially supported.

Resolution

  • Currently there is no resolution. VMware engineering is aware of this problem.
  • Subscribe to this article to be informed when the status changes.

Workaround:

Before attempting to redeploy a stateful host using Autodeploy, format the partitions on the boot device (BOOTBANK1, BOOTBANK2 and OSDATA) using external tools.

Powered by Wolken Software