"VMware vSphere Lifecycle Manager had an unknown error" when updating a 7.0 ESXi host using baselines

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Updating a 7.0 ESXi host using Lifecycle Manager baselines fails with the generic message "VMware vSphere Lifecycle Manager had an unknown error. Check the events and log files for details"
• Checking the /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log shows an error similar to the following followed by a backtrace (truncated backtrace shown below for brevity):

[YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[37856] [Originator@6876 sub=SingleHostRemediate] [hostUpdateBase 842] FaultMessage: Unknown MethodFault Error on Host: An error occurred while communicating with the remote host.

[YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[37856] [Originator@6876 sub=SingleHostRemediate] [hostUpdateBase 842] [backtrace begin] product: VMware Update Manager, version: 7.0.3, build: build-20983231, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release

--> backtrace[00] libvmacore.so[0x0037DA77]

• Checking the /var/run/log/esxupdate.log on the ESXi host for the most recent action as part of the update shows the opening of a vib from the vCenter Server repository for download but nothing further logged for that process id.
  
  [YYYY-MM-DDTHH:MM:SS] esxupdate: 2293615: downloader: INFO: Opening http://vcenter.example.com:9084/vum/repository/hostupdate/vmw/vib20/tools-light/VMware_locker_tools-light_12.1.0.20219665-20841705.vib for download

Where the above process id to be tracked in the esxupdate.log can be obtained from the task in the /var/run/log/hostd.log

For the example above with process id '2293615':

[YYYY-MM-DDTHH:MM:SS] info hostd[2101008] [Originator@6876 sub=Vimsvc.TaskManager opID=1815c9a4-85-5e91 user=vpxuser:com.vmware.vcIntegrity] Task Created : haTask-ha-host-vim.host.PatchManager.InstallV2-2780807994

[YYYY-MM-DDTHH:MM:SS] info hostd[2103399] [Originator@6876 sub=SysCommandPosix opID=1815c9a4-85-5e91 user=vpxuser:com.vmware.vcIntegrity] ForkExec(/usr/sbin/esxupdate) 2293615

• Checking the status of this process on the host, it is stuck in WAIT State, waiting on UPIPEW by running the following command live on the ESXi host replacing the process id as per the logs above (The /commands directory in an ESXi log bundle also has this file captured as ps_-cPTgjstz.txt):
  
  ps -cPTgjstz | grep -i 2293615

WID      CID      WorldName                           GID     PGID    SID     PCID     Type    State   Wait    CPU   Time          Command
2110250  2110250  python                              2110250  2100175  2100175  2100175  U       WAIT    UPIPEW  0-35      2.652627  /usr/bin/python /usr/sbin/esxupdate --HA

• Checking the process tree further, identify any zombie process under it by running the following command: 
  
  ps -Cc --tree | grep -i 2293615

x    mqO2100175  2100175  hostd                 hostd
x    tqO2110250  2110250  python                /usr/bin/python /usr/sbin/esxupdate --HA
x    mqO2100629  2100629  <zombie>
  
  Note: This may be found offline in an ESXi host log bundle under /commands/ps_-Cc---tree.txt.

Environment

VMware vCenter Server 7.0.x
VMware vSphere ESXi 7.0.x

Cause

This is a known issue which can occur on a system upgraded through a large number of versions and caused by a large number of old metadata files that are still present on the vCenter Server.

Resolution

This issue is resolved in vCenter Server 7.0 Update 3o (build number 21958407).

Workaround

If possible, a reset of the Update Manager database should resolve this issue. Refer to the following KB for details on this procedure: Resetting VMware Update Manager Database in vCenter Server Appliance 6.5/6.7/7.0/8.0

If a reset of the Update Manager database is not possible, attempt to delete the unwanted depots via dcli or API as this is not possible via the vSphere UI. Refer to Delete Depot Offline for more information.