Upgrading or Patching vCenter Server to 8.0 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

Upgrade pre-check on 8.0 fails with below error message :

---------------------------
Pre-upgrade check result

Error:
Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!

Resolution:
Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving the issue with Upgrading or Patching vCenter Server to 8.0 U2 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"

-------------------------

For migration from Windows vCenter Server, log file "C:\Users\%username%\AppData\Local\VMware\Migration-Assistant\logs.zip - CollectRequirements_com.vmware.eam_<DATE>.log" will show below error snippets :
[YYYY-MM-DDTHH:MM:SS]Z INFO eam.lib.eam-upgrade-prechecks Creating local EAM client(port=15005)
[YYYY-MM-DDTHH:MM:SS]Z ERROR eam Failed to execute trusted certificates check..
Traceback (most recent call last):
File "C:\migF629.tmp\PFiles\VMware\CIS\cis_upgrade_runner\payload\component-scripts\eam\lib\eam_ops.py", line 192, in retrieveUrls
eam.lib.pyEam.VmomiSupport.eam.fault.EamServiceNotInitialized: (eam.fault.EamServiceNotInitialized) {
msg = 'EAM is still loading from database. Please try again later.',
}
For upgrade from VCSA, log file /var/log/vmware/upgrade/CollectRequirements_com.vmware.eam_<DATE>.log will show below error snippets:

[YYYY-MM-DDTHH:MM:SS]Z INFO eam.lib.eam-upgrade-prechecks Creating local EAM client(port=15005)
[YYYY-MM-DDTHH:MM:SS]1Z ERROR eam Failed to execute trusted certificates check..
Traceback (most recent call last):
File "/tmp/vmware-upgrade-temp-dirOfZ3SyDImn/tmplnwe4R2iOv/payload/component-scripts/eam/lib/eam_ops.py", line 192, in retrieveUrls
allAgencies = esxAgentMgr.agency
eam.lib.pyEam.VmomiSupport.eam.fault.EamServiceNotInitialized: (eam.fault.EamServiceNotInitialized) {
msg = 'EAM is still loading from database. Please try again later.',
}

Patching VCSA from 8.x to 8.0 Update 2 will show below error message :

vCenter Server patching using CLI shows below errors

Command> software-packages install --staged

[YYYY-MM-DDTHH:MM:SS] : update is already staged. Proceeding to install.

[YYYY-MM-DDTHH:MM:SS] : Installing version: 8.0.2.00000
[YYYY-MM-DDTHH:MM:SS] : Running precheck ....
[YYYY-MM-DDTHH:MM:SS] : Installation failed. Retry to resume from the current state. Or please collect the VC support bundle.
Mismatch:
summary: Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!
resolution: Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving the issue with https://knowledge.broadcom.com/external/article?legacyId=94934.

If source VC is 6.x, eam log (/var/log/vmware/eam/eam.log or %ProgramData%/VMware/vCenterServer/logs/eam/eam.log) will show below entries:

[YYYY-MM-DDTHH:MM:SS]Z | INFO | vim-monitor | VcConnection.java | 199 | Connecting to vCenter as com.vmware.vim.eam extension
[YYYY-MM-DDTHH:MM:SS]Z | INFO | vim-monitor | VcConnection.java | 640 | Connecting to https://VC_IP/:8089/sdk/vimService via vCenter proxy http://localhost:80
[YYYY-MM-DDTHH:MM:SS]Z | ERROR | vim-monitor | VcConnection.java | 213 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.
[YYYY-MM-DDTHH:MM:SS]Z | WARN | vim-monitor | VcListener.java | 111 | Trying to recover from error
(vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
}

If source VC is 7.x or higher, eam log (/var/log/vmware/eam/eam.log) will show below entries:

Environment

VMware vCenter Server 8.0.x

Cause

As part of the EAM upgrade pre-checks, an EAM client is created to retrieve all EAM agencies and perform necessary SSL trust checks. This step might not succeed if an EAM client cannot be created because the EAM service is unable to log in to vCenter. This can occur due to a discrepancy between the "vpxd-extension" certificate stored in VECS and the certificate information stored in the vCenter Server Database for the EAM extension.

Resolution

Update the certificate for the Extensions in VPXD by following any of below options:

Option 1 - Update extensions using vCert script from vCert - expired certificate replacement script

Execute following command to update vpxd-extension thumbprint

./vCert.py --run config/manage_cert/op_manage-vc-ext-thumbprints.yaml
============================================================================================
Please enter a Single Sign-On administrator account [[email protected]]: <-- Press Enter to use the default
Please provide the password for [email protected]: <-- Enter the password for <[email protected]>

:::

------------------------!!! Attention !!!------------------------
Mismatched thumbprints detected.

Update extension thumbprints? [n]: y <-- Type 'y' and press Enter
============================================================================================
Restart eam service
service-control --restart eam

Option 2 - Update extensions using fixcerts script from Upgrading or Patching vCenter Server to 8.0 U2 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"

Update the extension's certificate using fixcerts script:

Download the attached fixcerts script
Copy the downloaded script to VCSA
Execute the script using below arguments to update the extensions

python fixcerts.py update --ExtensionType all

Sample screenshot:

Note:

eam, rbd & imagebuilder services will be restarted if it is in Running state. In above example, restart of rbd and Imagebuilder services are skipped as those services were in Stopped state before executing the script.
More details about fixcerts script available in How to Replace Expired Certificates on vCenter Server using Fixcerts Python Script

Option 3 - Update extensions with the following article, EAM "Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password" after replacing the vCenter Server certificates

Note:
If the issue persists after following the steps in the KB, please review the EAM logs and refer to the following article for further troubleshooting:
Upgrade to vCenter Server 8.0.x fails with "ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs"

Additional Information

The Japanese version of the pre-check error message corresponding to
"Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!"
is:
日本語の場合のエラーメッセージ：
"ソースの vSphere ESX Agent Manager (EAM) のアップグレードで、システムが信頼する証明書に照らしてチェックするための EAM URL を取得できませんでした。"

Attachments

fixcerts.py get_app