VCF ESXi Upgrade Fails at ESX_UPGRADE_VUM_STAGE_INSTALL_UPDATE due to Conflicting Third-Party PatchManager Task

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0
• VMware SDDC Manager

Issue/Introduction

VMware Cloud Foundation (VCF) ESXi upgrade fails in SDDC Manager.
The SDDC Manager LCM logs report a remediation failure:

Path: /var/log/vmware/vcf/lcm/lcm.log

"upgradeStatus": "COMPLETED_WITH_FAILURE",
"upgradeError": {
  "errorType": "RECOVERABLE",
  "stage": "ESX_UPGRADE_VUM_STAGE_INSTALL_UPDATE",
  "errorCode": "com.vmware.vcf.error.runtime.esx.update.vum.remediation.error.unexpected",
  "errorDescription": "Failed to install update due to unexpected error: VUM Remediate task failed"

The vpxa.log on the affected ESXi host indicates a conflict with another task locking the PatchManager:

Path: /var/run/log/vmware/vpxa.log

[VpxLRO] -- ERROR task-3 -- ######## -- patchManager -- vim.host.PatchManager.ScanV2: :vim.fault.TaskInProgress
--> (vim.fault.TaskInProgress) {
-->    task = 'vim.Task:haTask-ha-host-vim.host.PatchManager.InstallV2-550'
-->    msg = "Another task is already in progress."
--> }

vpxa.log also reveals which task is in progress:

 info vpxa[143578568] [Originator@6876 sub=Default opID=41b2ba1a-d4] [VpxLRO] -- ERROR task-19766 -- patchManager -- vim.host.PatchManager.InstallV2: vim.fault.PlatformConfigFault:
--> Result:
--> (vim.fault.PlatformConfigFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    text = "/usr/sbin/esxupdate returned with exit status: 14"
-->    msg = "An error occurred during host configuration."
--> }
--> Args:
-->
--> Arg metaUrls:
--> (string) [
-->    "/vmfs/volumes/#########/tsdmVib/metadata.zip"
--> ]
--> Arg bundleUrls:
--> (string) []
--> Arg vibUrls:
--> (string) [
-->    "/vmfs/volumes/##########/tsdmVib/DEL_bootbank_tsdm_19.22.0.3-1OEM.800.1.0.20613240.vib"
--> ]
--> Arg spec:
--> (vim.host.PatchManager.PatchManagerOperationSpec) {
-->    proxy = <unset>,
-->    port = <unset>,
-->    userName = <unset>,
-->    password = <unset>,
-->    cmdOption = <unset>

We can see the same error under ESXi --> Tasks

The esxupdate.log reveals continuous failed attempts to install a third-party VIB from a datastore:

Path: /var/run/log/vmware/esxupdate.log

esxupdate: ERROR: vmware.esximage.Errors.DependencyError: VIB DEL_bootbank_tsdm_19.22.0.3-1OEM.800.1.0.20613240 requires uwvmkcall-8.0.0-1, but the requirement cannot be satisfied within the ImageProfile.

Environment

VCF 5.x

ESXi 8.0.x

 

Cause

A third-party client (e.g., Dell PowerProtect/TSDM) is executing continuous automated attempts to install a VIB (DEL_bootbank_tsdm_19.22.0.3-1OEM.800.1.0.20613240.vib). This behavior creates an ongoing vim.host.PatchManager.InstallV2 task that locks the ESXi PatchManager, preventing the VCF upgrade vim.host.PatchManager.ScanV2 task from executing.

Resolution

 

• Disable the third-party automation, backup software (e.g., Transparent Snapshots Data Mover), or scripting targeting the ESXi hosts prior to the upgrade.
• Retry the ESXi upgrade workflow from the SDDC Manager UI.
• Once the VCF upgrade has completed successfully, re-enable the third-party automation.

Contact the third-party vendor for guidance on configuring proper maintenance windows or suppressing automated VIB deployments while infrastructure upgrades are in progress.