Upgrading or Patching vCenter Server to 8.0 U2 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Upgrade pre-check on 8.0 U2 fails with below error message :

---------------------------
Pre-upgrade check result
Error:
Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!
Resolution:
Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving the issue with https://kb.vmware.com/s/article/94934.
-------------------------

For migration from Windows vCenter Server, log file "C:\Users\%username%\AppData\Local\VMware\Migration-Assistant\logs.zip - CollectRequirements_com.vmware.eam_<DATE>.log" will show below error snippets :
2023-10-13T19:52:30.282Z INFO eam.lib.eam-upgrade-prechecks Creating local EAM client(port=15005)
2023-10-13T19:52:31.906Z ERROR eam Failed to execute trusted certificates check..
Traceback (most recent call last):
File "C:\migF629.tmp\PFiles\VMware\CIS\cis_upgrade_runner\payload\component-scripts\eam\lib\eam_ops.py", line 192, in retrieveUrls
eam.lib.pyEam.VmomiSupport.eam.fault.EamServiceNotInitialized: (eam.fault.EamServiceNotInitialized) {
msg = 'EAM is still loading from database. Please try again later.',
}
For upgrade from VCSA, log file /var/log/vmware/upgrade/CollectRequirements_com.vmware.eam_<DATE>.log will show below error snippets:

2023-10-13T20:10:09.417Z INFO eam.lib.eam-upgrade-prechecks Creating local EAM client(port=15005)
2023-10-13T20:10:09.441Z ERROR eam Failed to execute trusted certificates check..
Traceback (most recent call last):
File "/tmp/vmware-upgrade-temp-dirOfZ3SyDImn/tmplnwe4R2iOv/payload/component-scripts/eam/lib/eam_ops.py", line 192, in retrieveUrls
allAgencies = esxAgentMgr.agency
eam.lib.pyEam.VmomiSupport.eam.fault.EamServiceNotInitialized: (eam.fault.EamServiceNotInitialized) {
msg = 'EAM is still loading from database. Please try again later.',
}

Patching VCSA from 8.x to 8.0 Update 2 will show below error message :

vCenter Server patching using CLI shows below errors

Command> software-packages install --staged

[2023-10-04 09:38:00,042] : update is already staged. Proceeding to install.

[2023-10-04 09:39:12,836] : Installing version: 8.0.2.00000
[2023-10-04 09:39:43,914] : Running precheck ....
[2023-10-04 09:39:45,926] : Installation failed. Retry to resume from the current state. Or please collect the VC support bundle.
Mismatch:
summary: Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!
resolution: Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving the issue with https://kb.vmware.com/s/article/94934.

If source VC is 6.x, eam log (/var/log/vmware/eam/eam.log or %ProgramData%/VMware/vCenterServer/logs/eam/eam.log) will show below entries:

2023-10-17T11:42:57.195Z | INFO | vim-monitor | VcConnection.java | 199 | Connecting to vCenter as com.vmware.vim.eam extension
2023-10-17T11:42:57.236Z | INFO | vim-monitor | VcConnection.java | 640 | Connecting to https://vcsatest.vmware.com:8089/sdk/vimService via vCenter proxy http://localhost:80
2023-10-17T11:43:02.339Z | ERROR | vim-monitor | VcConnection.java | 213 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.
2023-10-17T11:43:02.339Z | WARN | vim-monitor | VcListener.java | 111 | Trying to recover from error
(vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
}

If source VC is 7.x or higher, eam log (/var/log/vmware/eam/eam.log) will show below entries:

Environment

VMware vCenter Server 8.0.x

Cause

As part of the EAM upgrade pre-checks, an EAM client is created to retrieve all EAM agencies and perform necessary SSL trust checks. This step might not succeed if an EAM client cannot be created because the EAM service is unable to log in to vCenter. This can occur due to a discrepancy between the "vpxd-extension" certificate stored in VECS and the certificate information stored in the vCenter Server Database for the EAM extension.

Resolution

Update the certificate for the Extensions in VPXD by following any of below options :
Note: Please follow Option 2 if source is Windows vCenter Server.

Option 1 - Update extensions using fixcerts script
Option 2 - Update extensions using article 318255

Update the extension's certificate using fixcerts script:

Download the attached fixcerts script
Copy the downloaded script to VCSA
Execute the script using below arguments to update the extensions

python fixcerts.py update --ExtensionType all

Sample screenshot:

Note:

Eam, rbd & imagebuilder services will be restarted if it is in Running state. In above example, restart of rbd and Imagebuilder services are skipped as those services were in Stopped state before executing the script.
More details about fixcerts script available in KB How to Replace Expired Certificates on vCenter Server using Fixcerts Python Script

Update the extension's certificate using KB2112577

Follow the steps in KB "Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password", ESX Agent Manager (com.vmware.vim.eam) solution user fails to log in after replacing the vCenter Server certificates in vCenter Server 6.x/7.x/8.x to update the individual extensions.

Attachments

fixcerts.py get_app