vSAN Disk Format Upgrade Hangs at 1% and Fails with Timeout Error in OSA Cluster

Products

• VMware vSAN 8.x

Issue/Introduction

• After upgrading vSphere from 7.x to 8.x, initiating a vSAN disk format upgrade (from version 15 to 20) triggers two tasks: Remediate vSAN Cluster and Convert Disk Format for vSAN. These tasks stall at 57% and 1% respectively, eventually failing with a timeout error.
  
  
  
• On the vCenter, /var/log/vmware/vsan-health/vsan-health.log file shows that the remediation step times out during task execution, causing the disk format conversion thread to abort:
  ERROR vsan-mgmt[1325571] [VsanClusterPrototypeImpl::WaitForUpdateTask opID=agw-0000269-6f53-W16862-W16864] Failed to wait for sub task task-1154567: (vim.fault.Timedout) {
  msg = 'Operation timed out.'

ERROR vsan-mgmt[08786] [VsanVcDiskFormatConverterImpl::_Run opID=agw-0000269-6f53] Disk conversion thread detcted vSAN error
Traceback (most recent call last):
  File "bora/vsan/diskmgmt/vpxd/pyMoVsan/VsanVcDiskFormatConverterImpl.py", line 4201, in _Run
  File "bora/vsan/diskmgmt/vpxd/pyMoVsan/VsanVcDiskFormatConverterImpl.py", line 4956, in ConvertDisks
  File "bora/vsan/diskmgmt/vpxd/pyMoVsan/VsanVcDiskFormatConverterImpl.py", line 4385, in _RemediateBeforeConversion
  File "bora/vsan/MgmtServer/VsanMgmtUtil.py", line 391, in WaitForTask
PyCppVmomi.vim.fault.VimFault: (vim.fault.VimFault) {
  msg = 'Cannot complete the operation. See the event log for details.',
  faultMessage = (vmodl.LocalizableMessage) [
    (vmodl.LocalizableMessage) {
      key = 'com.vmware.vsan.clustermgmt.msg.remediationfailed',
      message = 'Failed to remediate vSAN configuration.'
    }
  ]
}
INFO vsan-mgmt[08786] [VsanVcDiskFormatConverterImpl::_PostConversion opID=agw-0000269-6f53] Finished dfc for domain-###### result:(vim.cluster.VsanUpgradeStatusEx) {
  inProgress = false,
  aborted = true,
  completed = false,
  progress = 1,
  isPrecheck = false
} fault: (vim.fault.VimFault) {
  msg = 'Cannot complete the operation. See the event log for details.',
  faultMessage = (vmodl.LocalizableMessage) [
    (vmodl.LocalizableMessage) {
      key = 'com.vmware.vsan.clustermgmt.msg.remediationfailed',
      message = 'Failed to remediate vSAN configuration.'
    }
  ]
}

Environment

VMware vSAN 8.x

Cause

This issue is caused by stale metadata in the cluster configuration, specifically on vSAN OSA clusters where vSAN ESA is disabled. This stale metadata blocks the disk format upgrade process.

Resolution

Engage Broadcom Support for further assistance with this issue.