Failed to authenticate extension com.vmware.vim.eam to vCenter. Re-register missing EAM extension to vCenter

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• Our investigation revealed that com.vmware.vim.eam was absent from the Management Object Browser (MOB) page, and the vSphere ESX Agent Manager was not present within the vSphere UI (under Administration > Solutions > vCenter Server Extensions).
• EAM is no longer able to manage agent installs on ESXi hosts from vCenter or products connected to vCenter, such as vCLS VMs, NSX.
• The eam.log (/var/log/vmware/eam/eam.log) has entries similar to the following:
  
  
  

  YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 167 | Connecting to vCenter as com.vmware.vim.eam extension

  YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 603 | Connecting to https://vCenter_Server_FQDN:8089/sdk/vimService via vCenter proxy http://localhost:80

  YYYY-MM-DDTHH:MM:SS.MSZ | DEBUG | http-bio-0.0.0.0-15005-exec-1 | AllowAllSamlTokenPolicy.java | 24 | HealtStatus request's token subject name: machine-########-####-####-####-#######, subject domain: vsphere.local

  YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.

  YYYY-MM-DDTHH:MM:SS.MSZ | WARN | eam-0 | VcListener.java | 114 | Trying to recover from error

  (vim.fault.InvalidLogin) {

  faultCause = null,

  faultMessage = null

  }

  .

  .

  YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcListener.java | 121 | Retrying in 10

  or

  YYYY-MM-DDTHH:MM:SS.MSZ |  WARN | vim-async-1 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:################] Re-login failed, due to:
com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.
.
.
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password."

  YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | vim-monitor | VcListener.java | 124 | An unexpected error in the changes polling loop
com.vmware.eam.EamRemoteSystemException: Unexpected error communicating with the vCenter server.
.
.
Caused by: com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.


• The vpxd.log (/var/log/vmware/vpxd/vpxd.log)  has entries similar to the following:
  
  

   info vpxd[58229] [Originator@6876 sub=Default opID=######] [VpxLRO] -- ERROR lro-4##### -- SessionManager -- vim.SessionManager.loginExtensionByCertificate: vim.fault.InvalidLogin: 
  --> Result: 
  --> (vim.fault.InvalidLogin) { 
  -->    faultCause = (vmodl.MethodFault) null, 
  -->    faultMessage = <unset> 
  -->    msg = "" 
  --> } 
  --> Args: 
  --> 
  --> Arg extensionKey: 
  --> "com.vmware.vim.eam" 
  --> Arg locale: 
  --> 

  
  
  
  
• The below KBs to update the certificate of the EAM extension do not help: 
  • Upgrading or Patching vCenter Server to 8.0 U2 fails with precheck error "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"
  • How to Replace Expired Certificates on vCenter Server using Fixcerts Python Script(322249)
  • EAM "Failed to login to vCenter as extension 
    
    When attempting to update the EAM extension as per the above KBs the following error is seen:
    
    
    
    raise obj # pylint: disable-msg=E0702
    pyVmomi.VmomiSupport.NotFound: (vimfault.NotFound)
    msg = 'The object or item referred to could not be found.'

Environment

• VMware vCenter Server 8.0.x
• VMware vCenter Server 7.0.x

Cause

The EAM extension was accidentally unregistered from the vSphere HTML5 client.

Resolution

Note: Before undertaking the following steps, it is strongly recommended to create a snapshot of your vCenter Server, please refer KB: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

1. Connect to vCenter server via SSH with root access.
   
   
2. Stop EAM service:
   service-control --stop vmware-eam


3. Make sure the /etc/vmware-eam/firstboot/extension/extension.xml.installer file is correct:
   
   
   1. The "config - extension - certificate" XML tag has the following contents:
      <certificate>@CERT_PATH@</certificate>


   2. The "config - extension - health - url" XML tag has the following contents:
      <url>https://@HOST@:443/eam/eamService-web/health.xml</url>


   3. The "config - extension - servers - server - url" XML tag has the following contents:
      <url>https://@HOST@:443/eam/sdk/</url>
      
      
   4. The "config - extension - managedEntityInfos - managedEntityInfo - smallIconUrl" XML tag has the following contents:
      <smallIconUrl>http://@HOST@:80/eam/imx/esx_vm_16x.png</smallIconUrl>
      
      
   5. The "config - extension - managedEntityInfos - managedEntityInfo - iconUrl" XML tag has the following contents:
      <iconUrl>http://@HOST@:80/eam/imx/esx_vm_32x.png</iconUrl>
      
      
4. Make a backup of the file /usr/lib/vmware-eam/firstboot/eam_firstboot.py:
   cp /usr/lib/vmware-eam/firstboot/eam_firstboot.py /usr/lib/vmware-eam/firstboot/eam_firstboot.py.bak
   
   
5. Manipulate the file /usr/lib/vmware-eam/firstboot/eam_firstboot.py
   
   
   • Change the following lines -

   if fbAction == FBActions.FIRSTBOOT:
    fb.setup_config_files()
    fb.register_with_cm()
    fb.installService()
    fb.registerExtension()
    fb.setup_service_security()
    upgrade_dir = fb.getUpgradeDir()
    if(upgrade_dir):
       fb.importDepotTrustJson(upgrade_dir)
    startService(fb)


   To look like -

   if fbAction == FBActions.FIRSTBOOT:
    #fb.setup_config_files()
    #fb.register_with_cm()
    #fb.installService()
    fb.registerExtension()   <<< DO NOT COMMENT THIS LINE
    #fb.setup_service_security()
    #upgrade_dir = fb.getUpgradeDir()
    #if(upgrade_dir):
    #   fb.importDepotTrustJson(upgrade_dir)
    #startService(fb)

6. Execute the following command:
   PYTHONPATH=$PYTHONPATH:$VMWARE_PYTHON_PATH $VMWARE_PYTHON_BIN -B /usr/lib/vmware-eam/firstboot/eam_firstboot.py
   
   
7. EAM extension should be registered properly to the vCenter Server. Check the MOB page to confirm the eam extension is present.
   
   
8. Check the vpx_ext table to validate if the eam extension (com.vmware.vim.eam) is present:
   psql -U postgres -d VCDB -c "SELECT * FROM vpx_ext;"


9. Ensure eam's service registration is present:
   /usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert | grep -i "Service Type:" | sort | uniq -c
   
   
10. Revert the changes done to the eam_firstboot.py file. 
    

11. Start EAM service:
    service-control --start vmware-eam 

Additional Information