Error: "Cannot find vSphere HA master agent" and Looping HA Configuration Tasks After vCenter Server Upgrade

Products

• VMware vSphere ESXi
• VMware vCenter Server 8.0

Issue/Introduction

• The host HA state reports as "Uninitialized".
  
  
• vCenter displays the error: "Cannot find vSphere HA master agent.
  
  
• Network connectivity (port 8182) and DNS resolution to the vCenter FQDN are fully functional.
  
  
• Checking the host packages via CLI shows the vmware-fdm VIB is stuck at the older version.
  
  
• In the ESX host, /var/run/log/esxupdate.log contains the message:
  
  LiveImageInstaller: DEBUG: Only reboot-required installations are possible right now as a reboot-required installation has been done.

• In the Esxi host, /var/run/log/fdm.log we see below error
  
  YYYY-MM-DDTHH:MM:SS In(14) [+] esxupdate [34953784] : VMware_bootbank_vmware-fdm_8.0.3-25600417.vib for download
YYYY-MM-DDTHH:MM:SS In (14) esxupdate [34953784] : Successfully executed cmd: localcli -- plugin-dir /usr/lib/vmware/esxcli/int/ networkinternal firewall firewallRuleset set -- ruleset-id esxupdate -- service-name esxupdate -- enabled false
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate [34953784] : File "/lib64/python3.11/site-packages/vmware/esximage/Downloader.py", line 487, in _openfromurl
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate [34953784] : return self ._ retry(self ._ open_for_download, self.options['retry' ])
YYYY-MM-DDTHH:MM:SS Er(11) esxupdate[34953784] : File "/1ib64/python3.11/site-packages/vmware/esximage/Downloader.py", line 228, in retry
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate [34953784] : return opener. open (url, timeout=self. options ['timeout' ] )
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate[34953784] : ValueError: unknown url type: '/tmp/\nVMware_bootbank_vmware-fdm_8.0.3-25600417.vib'
YYYY-MM-DDTHH:MM:SS Er(11) esxupdate[34953784]: Traceback (most recent call last) :
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate[34953784] : main ()
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate [34953784] : File "/usr/lib/vmware/esxcli-software", line 1122, in main
YYYY-MM-DDTHH:MM:SS Er(11) esxupdate[34953784]: ret = CMDTABLE [command] (options)
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate[34953784] : livePatcher = Transaction. LivePatcherMount. InitVibInstall (
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate[34953784] : File "/lib64/python3.11/site-packages/vmware/esximage/Transaction.py", line 2602, in InitVibInstall
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate [34953784] : Transaction. GetVibsAndMetaFromSources (vibUrls, metaUrls, vibSpecs,
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate [34953784] : vibsToInstall = Transaction ._ getVibsFromUrls (vibUrls)
YYYY-MM-DDTHH:MM:SSZ Er (11) esxupdate [34953784] : File "/lib64/python3.11/site-packages/vmware/esximage/Transaction.py", line 1000, in getVibsFromUrls
YYYY-MM-DDTHH:MM:SS Er (11) esxupdate [34953784] : raise Errors. VibDownloadError (url, None, str (e))
YYYY-MM-DDTHH:MM:SS Er(11) esxupdate[34953784]: vmware.esximage.Errors.VibDownloadError: ('/tmp/\nVMware_bootbank_vmware-fdm_8.0.3-25600417.vib', None, '(\'/tmp/\\nVMware_bootbank_vmware-fdm_8.0.3-25600417.vib\', \'\', "unknown url type\'/tmp/\\\\nVMware_bootbank_vmware-fdm_8.0.3-25600417.vib\'")

• While installing FDM vib manually we get error as "Host is not changed. Reboot is pending from previous transaction."

  root@######] esxcli software vib install -v /tmp/VMware_bootbank_vmware-fdm_8.0.3-25600417.vib
Installation Result
Message: Host is not changed. Reboot is pending from previous transaction.
VIBs Installed:
VIBs Removed:
VIBs Skipped: VMware_bootbank_vmware-fdm_8.0.3-25600417
Reboot Required: true
DPU Results:

Environment

vSphere 8.x

Cause

• This issue occurs when the affected ESXi hosts have a pending reboot from a previous system change, such as staging a patch, updating firmware, or installing a driver.
• When an ESXi host is in a "pending reboot" state, the local package manager (esxupdate) places a strict lock on live installations. This lock silently blocks vCenter from dynamically uninstalling the old Fault Domain Manager (vmware-fdm) agent and pushing the new version required to communicate with the upgraded vCenter. Because the installation fails in the background and rolls back, the HA cluster election never completes.

Resolution

To clear the live-installation lock and allow vCenter to update the HA agent, the affected hosts must be rebooted.

1. In the vSphere Client, navigate to the Cluster > Configure > vSphere Availability.
   
   
2. Click Edit, uncheck the box to disable vSphere HA, and click OK. Wait for the "Unconfigure HA" tasks to complete.
   
   
3. Migrate any running virtual machines off the affected ESXi hosts and place them into Maintenance Mode.
   
   
4. Restart each of the affected ESXi hosts. This will formally clear the esxupdate pending-reboot lock.
   
   
5. Once the hosts are back online and connected to vCenter, take them out of Maintenance Mode.
   
   
6. Return to your Cluster > Configure > vSphere Availability settings and enable vSphere HA.