ESX Upgrade Task stuck for hours at PostCheck task

Products

• VMware vSphere ESXi

Issue/Introduction

• Upgrade was triggered via Operations for ESXi server from 9.0.2 to 9.1

• The Upgrade is stuck at PostCheck step for hours. Similar error will see in UI:
  ESX upgrade using vSphere Lifecycle Manager images failed. Error: Error in vLCM. Connection to vCenter failed.
ESX vLCM Upgrade failed. Contact Broadcom Technical Support for assistance. Check for errors in the lcm log files located on SDDC Manager under /tmp/lcm.
com.vmware.vcf.error.runtime.esx.vlcm.upgrade.failed.exception

• The vCenter UI shows the ESX hosts finished with the upgrade

• The VCF Operations UI shows the task ongoing

• The SDDC Manager UI shows the task also incomplete

• The SDDC Manager log files shows an ERROR "completed post processing cleanups"

Environment

SDDC Manager 9.1.x

Cause

• Since EsxCommandExecutor initialization happened once at the first usage but cannot be initialized, every time ESX postcheck updates known_hosts for ESX host it throw Error not Exception, because EsxCommandExecutor is not initialized.
• This is why no exception is logged in lcm-debug.log, as the code catch Exception, Error is NOT covered. 
• The above race condition normally does not happen, but once it happens, it cannot recover, and service restart is the only option to recover

Resolution

1. Restart lcm service from SDDC Manager
   • SSH to SDDC Manager and run below command

   • systemctl restart lcm