Upgrading or Patching vCenter Server to 8.0 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"
search cancel

Upgrading or Patching vCenter Server to 8.0 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"

book

Article ID: 344775

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

  • The vCenter Server 8.0 upgrade pre-check fails with the following error message:
Error: Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!

Resolution: Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving the issue with Upgrading or Patching vCenter Server to 8.0 U2 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System" 
 
  • For migration from Windows vCenter Server, log file "C:\Users\%username%\AppData\Local\VMware\Migration-Assistant\logs.zip - CollectRequirements_com.vmware.eam_<DATE>.log" will show the following error snippets:

    [YYYY-MM-DDTHH:MM:SS]Z INFO eam.lib.eam-upgrade-prechecks Creating local EAM client(port=15005)
    [YYYY-MM-DDTHH:MM:SS]Z ERROR eam Failed to execute trusted certificates check..
    Traceback (most recent call last):
      File "C:\####.tmp\PFiles\VMware\CIS\cis_upgrade_runner\payload\component-scripts\eam\lib\eam_ops.py", line ###, in retrieveUrls
    eam.lib.pyEam.VmomiSupport.eam.fault.EamServiceNotInitialized: (eam.fault.EamServiceNotInitialized) {
       msg = 'EAM is still loading from database. Please try again later.',
    }

  • During a vCenter Server upgrade, the log file /var/log/vmware/upgrade/CollectRequirements_com.vmware.eam_<DATE>.log contains the following error snippets:
[YYYY-MM-DDTHH:MM:SS]Z INFO eam.lib.eam-upgrade-prechecks Creating local EAM client(port=15005)
[YYYY-MM-DDTHH:MM:SS]1Z ERROR eam Failed to execute trusted certificates check..
Traceback (most recent call last):
  File "/tmp/vmware-upgrade-temp-dir##########/tmp#########/payload/component-scripts/eam/lib/eam_ops.py", line ###, in retrieveUrls
    allAgencies = esxAgentMgr.agency
eam.lib.pyEam.VmomiSupport.eam.fault.EamServiceNotInitialized: (eam.fault.EamServiceNotInitialized) {
   msg = 'EAM is still loading from database. Please try again later.',
}
  • Patching the vCenter Server from 8.x to 8.0 Update 2 results in the following error message:
  • The following errors appear during vCenter Server patching via the CLI:
Command > software-packages install --staged

[YYYY-MM-DDTHH:MM:SS] : update is already staged. Proceeding to install.

[YYYY-MM-DDTHH:MM:SS] : Installing version: 8.0.2.00000
[YYYY-MM-DDTHH:MM:SS] : Running precheck ....
[YYYY-MM-DDTHH:MM:SS] : Installation failed. Retry to resume from the current state. Or please collect the VC support bundle.
Mismatch:
summary: Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!
resolution: Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving the issue with https://knowledge.broadcom.com/external/article?legacyId=94934.
  • If the source vCenter Server is running version 6.x, the eam.log (/var/log/vmware/eam/eam.log or %ProgramData%\VMware\vCenterServer\logs\eam\eam.log) will show the following entries:
[YYYY-MM-DDTHH:MM:SS]Z |  INFO | vim-monitor | VcConnection.java | ### | Connecting to vCenter as com.vmware.vim.eam extension
[YYYY-MM-DDTHH:MM:SS]Z |  INFO | vim-monitor | VcConnection.java | ### | Connecting to https://VC_IP/:8089/sdk/vimService via vCenter proxy http://localhost:80
[YYYY-MM-DDTHH:MM:SS]Z | ERROR | vim-monitor | VcConnection.java | ### | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.
[YYYY-MM-DDTHH:MM:SS]Z |  WARN | vim-monitor | VcListener.java | ### | Trying to recover from error
(vim.fault.InvalidLogin) {
   faultCause = null,
   faultMessage = null
}
  • If the source vCenter Server is version 7.x or later, the eam.log (/var/log/vmware/eam/eam.log) will show the following entries:
[YYYY-MM-DDTHH:MM:SS]Z |  INFO | vim-monitor | OpId.java | ## | [vim:loginExtensionByCertificate:################] created from [Retry:Login:com.vmware.vim.eam:################]
[YYYY-MM-DDTHH:MM:SS]Z |  INFO | vim-async-# | OpIdLogger.java | ## | [vim:loginExtensionByCertificate:################] Failed.
[YYYY-MM-DDTHH:MM:SS]Z |  WARN | vim-async-# | ExtensionSessionRenewer.java | ### | [Retry:Login:com.vmware.vim.eam:################] Re-login failed, due to: com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

As part of the EAM upgrade pre-checks, an EAM client is created to retrieve all EAM agencies and perform necessary SSL trust checks. This step might not succeed if an EAM client cannot be created because the EAM service is unable to log in to vCenter. This can occur due to a discrepancy between the "vpxd-extension" certificate stored in VECS and the certificate information stored in the vCenter Server Database for the EAM extension.

Resolution

This issue has been resolved in vCenter version 8.0 Update 3g

Workaround: 

Update the certificate for the Extensions in VPXD by following any of below options: 

Note: Ensure there are valid offline snapshots of the linked vCenter Servers or in case of standalone vCenter Server, take a snapshot without memory.

Option 1 - Update extensions using vCert script from vCert - expired certificate replacement script

    1. Execute the following command to update vpxd-extension thumbprint.

      ./vCert.py --run config/manage_cert/op_manage-vc-ext-thumbprints.yaml
      ============================================================================================
      Please enter a Single Sign-On administrator account [[email protected]]: <-- Press Enter to use the default
      Please provide the password for [email protected]: <-- Enter the password for <[email protected]>

      :::

      ------------------------!!! Attention !!!------------------------
      Mismatched thumbprints detected.

      Update extension thumbprints? [n]: y   <-- Type 'y' and press Enter
      ============================================================================================

      Note: If a custom vSphere domain name is configured, specify the custom name.

    2. Restart the vmware-eam service by running the command : service-control --restart vmware-eam

Option 2 - Run the attached fixcerts script to update the extensions.

    1. Download the attached fixcerts script.
    2. Copy the downloaded script to vCenter Server.
    3. Execute the script using the below arguments to update the extensions.
python fixcerts.py update --ExtensionType all

Sample screenshot:

 
Note:
 

Additional Information

The Japanese version of the pre-check error message corresponding to "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!" is: "日本語の場合のエラーメッセージ："ソースの vSphere ESX Agent Manager (EAM) のアップグレードで、システムが信頼する証明書に照らしてチェックするための EAM URL を取得できませんでした。"

Attachments

fixcerts.py get_app
Powered by Wolken Software