Error: "Failed to authenticate extension com.vmware.vim.eam to vCenter. Re-register missing EAM extension to vCenter" during vCenter update precheck

Products

• VMware vCenter Server

Issue/Introduction

• Attempting to patch vCenter Server via VAMI fails during precheck with the following error: Failed to authenticate extension com.vmware.vim.eam to vCenter. Re-register missing EAM extension to vCenter.
  
  
• The com.vmware.vim.eam extension is missing from the Management Object Browser (MOB) and the vSphere ESX Agent Manager does not appear in the vSphere Client UI under Administration > Solutions > vCenter Server Extensions.
  
  
• The ESX Agent Manager (EAM) fails to manage agent installations on ESXi hosts. This disrupts tasks originating from vCenter Server and integrated solutions, including vSphere Cluster Services (vCLS) and NSX.
  
  
• The following entries are recorded in the /var/log/vmware/eam/eam.log file on the vCenter Server:
  YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 167 | Connecting to vCenter as com.vmware.vim.eam extension
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 603 | Connecting to https://vCenter_Server_FQDN:8089/sdk/vimService via vCenter proxy http://localhost:80
YYYY-MM-DDTHH:MM:SS.MSZ | DEBUG | http-bio-0.0.0.0-15005-exec-1 | AllowAllSamlTokenPolicy.java | 24 | HealtStatus request's token subject name: machine-########-####-####-####-#######, subject domain: vsphere.local
YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.
YYYY-MM-DDTHH:MM:SS.MSZ | WARN | eam-0 | VcListener.java | 114 | Trying to recover from error
(vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
}
.
.
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcListener.java | 121 | Retrying in 10
or
YYYY-MM-DDTHH:MM:SS.MSZ |  WARN | vim-async-1 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:################] Re-login failed, due to:
com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.
.
.
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password."
YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | vim-monitor | VcListener.java | 124 | An unexpected error in the changes polling loop
com.vmware.eam.EamRemoteSystemException: Unexpected error communicating with the vCenter server.
.
.
Caused by: com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.
  
  
• The following entries were observed in the /var/log/vmware/vpxd/vpxd.log of the vCenter Server:
  YYYY-MM-DDTHH:MM:SS.MSZ info vpxd[58229] [Originator@6876 sub=Default opID=######] [VpxLRO] -- ERROR lro-4##### -- SessionManager -- vim.SessionManager.loginExtensionByCertificate: vim.fault.InvalidLogin: 
--> Result: 
--> (vim.fault.InvalidLogin) { 
-->    faultCause = (vmodl.MethodFault) null, 
-->    faultMessage = <unset> 
-->    msg = "" 
--> } 
--> Args: 
--> 
--> Arg extensionKey: 
--> "com.vmware.vim.eam" 
--> Arg locale:
  
  
• The issue persists even after attempting to update the EAM extension certificate using the following KBs:
  EAM "Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password" after replacing the vCenter Server certificates
  Replace certificates on vCenter server using the Fixcerts script
  Upgrading or Patching vCenter Server to 8.0 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"
  When attempting to update the EAM extension as per the above KBs, the following error is seen:
  raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.NotFound: (vimfault.NotFound)
msg = 'The object or item referred to could not be found.'
  
  
  
• EAM extension does not exist in Postgres database, which is confirmed by running: /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "select ext_id,thumbprint from vpx_ext where ext_id='com.vmware.vim.eam';"

Environment

vCenter Server 8.x
vCenter Server 7.x

Cause

The EAM extension was unregistered from the vSphere Client.

Resolution

Note: Before undertaking the following steps, it is strongly recommended to create a snapshot of the vCenter Server virtual machine. For more information, refer to KB: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

1. Connect to vCenter Server via SSH with root access.
   
   
2. Stop EAM service:
   service-control --stop vmware-eam


3. Make sure the /etc/vmware-eam/firstboot/extension/extension.xml.installer file is correct:
   
   
   1. The "config - extension - certificate" XML tag has the following contents:
      <certificate>@CERT_PATH@</certificate>


   2. The "config - extension - health - url" XML tag has the following contents:
      <url>https://@HOST@:443/eam/eamService-web/health.xml</url>


   3. The "config - extension - servers - server - url" XML tag has the following contents:
      <url>https://@HOST@:443/eam/sdk/</url>
      
      
   4. The "config - extension - managedEntityInfos - managedEntityInfo - smallIconUrl" XML tag has the following contents:
      <smallIconUrl>http://@HOST@:80/eam/imx/esx_vm_16x.png</smallIconUrl>
      
      
   5. The "config - extension - managedEntityInfos - managedEntityInfo - iconUrl" XML tag has the following contents:
      <iconUrl>http://@HOST@:80/eam/imx/esx_vm_32x.png</iconUrl>
      
      
4. Create a backup of the /usr/lib/vmware-eam/firstboot/eam_firstboot.py file:
   cp /usr/lib/vmware-eam/firstboot/eam_firstboot.py /usr/lib/vmware-eam/firstboot/eam_firstboot.py.bak
   
   
5. Modify the /usr/lib/vmware-eam/firstboot/eam_firstboot.py file to include the necessary changes.
   

   Before:

   if fbAction == FBActions.FIRSTBOOT:
    fb.setup_config_files()
    fb.register_with_cm()
    fb.installService()
    fb.registerExtension()
    fb.setup_service_security()
    upgrade_dir = fb.getUpgradeDir()
    if(upgrade_dir):
       fb.importDepotTrustJson(upgrade_dir)
    startService(fb)

   
   After:

   if fbAction == FBActions.FIRSTBOOT:
    #fb.setup_config_files()
    #fb.register_with_cm()
    #fb.installService()
    fb.registerExtension()   <<< DO NOT COMMENT THIS LINE
    #fb.setup_service_security()
    #upgrade_dir = fb.getUpgradeDir()
    #if(upgrade_dir):
    #   fb.importDepotTrustJson(upgrade_dir)
    #startService(fb)

6. Execute the following command to re-register the EAM extension:
   PYTHONPATH=$PYTHONPATH:$VMWARE_PYTHON_PATH $VMWARE_PYTHON_BIN -B /usr/lib/vmware-eam/firstboot/eam_firstboot.py


7. The EAM extension should now be successfully registered to the vCenter Server. To confirm, verify that the com.vmware.vim.eam extension is present on the Management Object Browser (MOB) page.
   
   
8. Check the vpx_ext table to validate if the eam extension (com.vmware.vim.eam) is present:
   psql -d VCDB -U postgres -c "select ext_id,thumbprint from vpx_ext where ext_id='com.vmware.vim.eam';"
   
   
9. Ensure EAM's service registration is present:
   /usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert | grep -i "Service Type: cs.eam" | sort | uniq -c
   
   
10. Revert the changes done to the eam_firstboot.py file. 
    
    
11. Start the EAM service:
    service-control --start vmware-eam