vCenter Server update is failing the Pre-Update Check with the error: Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!

Products

VMware vCenter Server

Issue/Introduction

Updating vCenter Server from the VAMI, the pre-check error below is shown:

---------------------------
Pre-update check result
Error:
Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!
Resolution:
Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving the issue with https://kb.vmware.com/s/article/94934.
-------------------------

Checking the /var/log/vmware/applmgmt/PatchRunner.log, you may see the following or similar logging:

2024-06-24T20:20:44.12Z eam:CollectRequirements INFO eam.lib.eam-upgrade-prechecks Creating local EAM client(port=15005)
2024-06-24T20:20:44.26Z eam:CollectRequirements ERROR eam Failed to execute trusted certificates check..
Traceback (most recent call last):
File "/storage/core/software-update/updates/8.0.2.00400/scripts/patches/payload/components-script/eam/lib/eam_ops.py", line 192, in retrieveUrls
allAgencies = esxAgentMgr.agency
File "/storage/core/software-update/updates/8.0.2.00400/scripts/patches/payload/components-script/eam/lib/pyEam/VmomiSupport.py", line 592, in __call__
return self.f(*args, **kwargs)
File "/storage/core/software-update/updates/8.0.2.00400/scripts/patches/payload/components-script/eam/lib/pyEam/VmomiSupport.py", line 397, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/storage/core/software-update/updates/8.0.2.00400/scripts/patches/payload/components-script/eam/lib/pyEam/StubAdapterAccessorImpl.py", line 43, in InvokeAccessor
return self.InvokeMethod(mo, info, (prop, ))
File "/storage/core/software-update/updates/8.0.2.00400/scripts/patches/payload/components-script/eam/lib/pyEam/SoapAdapter.py", line 1577, in InvokeMethod
raise obj # pylint: disable-msg=E0702
eam.lib.pyEam.VmomiSupport.eam.fault.EamServiceNotInitialized: (eam.fault.EamServiceNotInitialized) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'EAM is still loading from database. Please try again later.',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}

Checking in /var/log/vmware/eam/eam.log, you may see the following or similar logging repeating:

Checking in the vSphere Client UI, you may see no EAM agency listed in Administration>vCenter Server Extensions>ESX Agent Manager

Following the steps in: Upgrading or Patching vCenter Server to 8.0 U2 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System" and its further linked KB articles does not provide a resolution allowing you to proceed past the Pre-Update Check and update the vCenter Server.

Environment

vCenter Server 8.x

Cause

As part of the EAM upgrade pre-checks, an EAM client is created to retrieve all EAM agencies and perform necessary SSL trust checks. This step might not succeed if an EAM client cannot be created because the EAM service is unable to log in to vCenter (this can occur due to a discrepancy between the "vpxd-extension" certificate stored in VECS and the certificate information stored in the vCenter Server Database for the EAM extension), or if there is an issue with any of the EAM agencies listed in the VCDB.

This environment previously had NSX Trend Micro used, and when removing that from the environment an old/stale agency reference was left over in the VCDB.

Placing the cluster in Retreat Mode does not cleanup/remove vCLS VMs

How to Disable vCLS on a Cluster via Retreat Mode

Resolution

Place the cluster in Retreat Mode (does not cleanup/remove vCLS VMs as EAM service is having an issue)

How to Disable vCLS on a Cluster via Retreat Mode

Then, in an SSH session to the vCenter Server VCSA VM, log into the vCenter Server VCDB:

/opt/vmware/vpostgres/current/bin/psql -U postgres VCDB

- List the Agencies, find the Agency from the error in eam.log above (our example in the above eam.log was Agency ID xxxxxxxx-xxxx-xxxx-xxxx-exampleID showing the NSX Trend Micro EAM Agency leftover and not in use anymore)
  
  select * from vpx_ext_data:
- Delete the entry from the vcdb vpx_ext_data table that corresponds with the agency ID
  
  delete from vpx_ext_data where data_key = 'data_key from Agency found with above command';
  - NOTE: replace the 'data_key from Agency found with above command' with the actual data_key value from your Agency shown to be at issue.
  - To confirm, run the select command from above to see that the Agency at issue is no longer listed
- Exit the VCDB
  
  \q
Restart the EAM service

service-control --restart eam
- vCLS VMs are deleted from the cluster as the it is still in Retreat Mode
Exit the cluster from Retreat Mode (See Retrea Mode link above)
- vCLS VMs redeployed in the cluster
Pre-Update Check no longer failing in the VAMI with EAM and its logging are back to normal with new vCLS VMs deployed

Additional Information

Upgrading or Patching vCenter Server to 8.0 U2 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"