EAM "Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password" after replacing the vCenter Server certificates
search cancel

EAM "Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password" after replacing the vCenter Server certificates

book

Article ID: 318255

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

After replacing certificates on vCenter Server, the following symptoms may appear:

  • ESX Agent Manager (EAM) solution user fails to log in vCenter server (vpxd).
  • In the /var/log/vmware/eam/eam.log (on VCSA) file:
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 167 | Connecting to vCenter as com.vmware.vim.eam extension
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 603 | Connecting to https://vCenter_Server_FQDN:8089/sdk/vimService via vCenter proxy http://localhost:80
YYYY-MM-DDTHH:MM:SS.MSZ | DEBUG | http-bio-0.0.0.0-15005-exec-1 | AllowAllSamlTokenPolicy.java | 24 | HealtStatus request's token subject name: machine-########-####-####-####-#######, subject domain: vsphere.local
YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.
YYYY-MM-DDTHH:MM:SS.MSZ | WARN | eam-0 | VcListener.java | 114 | Trying to recover from error
(vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
}
at sun.reflect.GeneratedConstructorAccessor82.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173)
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:31)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:141)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:102)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:89)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:84)
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:112)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:273)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:144)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:77)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182)
at com.sun.proxy.$Proxy48.loginExtensionByCertificate(Unknown Source)
at com.vmware.eam.vc.VcConnection.connectEam(VcConnection.java:171)
at com.vmware.eam.vc.VcListener.login(VcListener.java:149)
at com.vmware.eam.vc.VcListener.main(VcListener.java:129)
at com.vmware.eam.vc.VcListener.call(VcListener.java:111)
at com.vmware.eam.vc.VcListener.call(VcListener.java:60)
at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35)
at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcListener.java | 121 | Retrying in 10

  • Unable to deploy VIBs to ESXi hosts from NSX for vSphere or vCloud Networking and Security.
  • vCenter Server experiences high CPU usage.
  • Error regarding DRS functionality:

    vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs

Environment

VMware vCenter Server 6.x
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This issue can happen:
  • There is a mismatch between vpxd-extension certificate stored in VECS and the certificate information stored in vCenter Server Database for EAM extension.
  • The EAM extension is missing in vSphere Web Client. For this scenario, call Technical Support and refer the Reregister EAM extension to vCenter.

Resolution

This issue is fixed in vCenter server 8.0 U3.

To resolve the issue, update the vpxd-extension certificate of vCenter Server by following below steps.

Process to update vpxd-extension certificate of vCenter Server Appliance:

  1. Log in to the vCenter Server Appliance using SSH. 
  2. Run this command to retrieve the vpxd-extension solution user certificate and key:

    mkdir /certificate

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
     
  3. Edit the "<vCenter_Server_Hostname>" in the below command and run this command to update the extension's certificate with vCenter Server.

    python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <vCenter_Server_Hostname> -u [email protected]

    Note: The default user and domain is [email protected]. Change the domain to match the environment's vCenter SSO. When prompted, type in the [email protected] password.

  4. Restart the VMware ESX Agent Manager service with these commands: 

    service-control --stop vmware-eam
    service-control --start vmware-eam

 
 



Additional Information

In certain situations, the following error may appear:"certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'". This error can be safely ignored if the error appears after the message "Successfully updated certificate for "com.vmware.vim.eam" extension" as this message confirms that Extension certificate updated successfully with vCenter Server.

root@hostname [ ~ ]# python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <vCenter_Server_Hostname> -u [email protected]
Password to connect to VC server for user="[email protected]":
2021-03-11T22:31:22.517Z  Updating certificate for "com.vmware.vim.eam" extension
2021-03-11T22:31:22.649Z  Successfully updated certificate for "com.vmware.vim.eam" extension
Traceback (most recent call last):
  File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 175, in <module>
    update_extension_cert_in_VC()
  File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 163, in update_extension_cert_in_VC
    sessionMgr = si.content.sessionManager
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 577, in __call__
    return self.f(*args, **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 382, in _InvokeAccessor
    return self._stub.InvokeAccessor(self, info)
  File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
    return self.InvokeMethod(mo, info, (prop, ))
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1525, in InvokeMethod
    conn.request('POST', self.path, req, headers)
  File "/usr/lib/python3.7/http/client.py", line 1277, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1323, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1272, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1032, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 972, in send
    self.connect()
  File "/usr/lib/python3.7/http/client.py", line 1447, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'. (_ssl.c:1076)


Other similar KB articles:
"WARNING: VMware ESX Agent Manager may have failed to start", EAM Service fails to start after vCenter Server reboot
"Internal error occurs during vSphere ESX Agent Manager pre-upgrade checks" upgrading the vCenter Server Appliance
Powered by Wolken Software