VCSA pre-check upgrade failing in stage 2 when upgrading from 7.0 U2 to 8.0 U3 with error "Error Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!"
Article ID: 416935
Updated On:
Products
Issue/Introduction
- Upgrade pre-check on 8.0 fails with below error message :
---------------------------
Pre-upgrade check result
Error Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted
certificates by the System!
Resolution
Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance
and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving
the issue with https://kb.vmware.com/s/article/94934.
From VCSA logs
/var/log/vmware/upgrade/CollectRequirements_com.vmware.eam_2025_10_20_12_05.log
ERROR eam Failed to execute trusted certificates check..
Traceback (most recent call last):
File "/tmp/vmware-upgrade-temp-dirsYkewi89rY/tmpFNBSZeKZmz/payload/component-scripts/eam/lib/eam_ops.py", line 192, in retrieveUrls
allAgencies = esxAgentMgr.agency
File "/tmp/vmware-upgrade-temp-dirsYkewi89rY/tmpFNBSZeKZmz/payload/component-scripts/eam/lib/pyEam/VmomiSupport.py", line 612, in __call__
return self.f(*args, **kwargs)
File "/tmp/vmware-upgrade-temp-dirsYkewi89rY/tmpFNBSZeKZmz/payload/component-scripts/eam/lib/pyEam/VmomiSupport.py", line 400, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/tmp/vmware-upgrade-temp-dirsYkewi89rY/tmpFNBSZeKZmz/payload/component-scripts/eam/lib/pyEam/StubAdapterAccessorImpl.py", line 44, in InvokeAccessor
return self.InvokeMethod(mo, info, (prop, ))
File "/tmp/vmware-upgrade-temp-dirsYkewi89rY/tmpFNBSZeKZmz/payload/component-scripts/eam/lib/pyEam/SoapAdapter.py", line 1607, in InvokeMethod
raise obj # pylint: disable-msg=E0702
eam.lib.pyEam.VmomiSupport.eam.fault.EamServiceNotInitialized: (eam.fault.EamServiceNotInitialized) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'EAM is still loading from database. Please try again later.',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/vmware-upgrade-temp-dirsYkewi89rY/tmpFNBSZeKZmz/payload/component-scripts/eam/__init__.py", line 119, in _trustCertsCheck
trustedCheckResult = urlTrustCheck(context)
File "/tmp/vmware-upgrade-temp-dirsYkewi89rY/tmpFNBSZeKZmz/payload/component-scripts/eam/lib/eam-upgrade-prechecks.py", line 58, in urlTrustCheck
_TRUST_FILE
File "/tmp/vmware-upgrade-temp-dirsYkewi89rY/tmpFNBSZeKZmz/payload/component-scripts/eam/lib/eam_ops.py", line 210, in retrieveUrls
raise Exception('Cannot retrieve EAM agencies', exc)
Exception: ('Cannot retrieve EAM agencies', (eam.fault.EamServiceNotInitialized) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'EAM is still loading from database. Please try again later.',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
})
INFO extensions The component script returned '{'coreRequirement': {'requiredSrcDiskSpace': 0.0, 'requiredDstDiskSpace': {}, 'exportEstimationTime': 0, 'importEstimationTime': 0}, 'dstPortSpecs': [], 'srcPorts': [], 'userOptionSpecs': [], 'installArguments': {}, 'extraArguments': {}, 'requirementMismatchSpecs': [{'text': Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!, 'description': None, 'resolution': Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving the issue with https://kb.vmware.com/s/article/94934., 'problemId': None, 'severity': 'ERROR'}]}'
(END)
/var/log/vmware/eam/eam.log
YYYY-MM-DDTHH:MM:SS | INFO | vim-monitor | ExtensionSessionRenewer.java | 190 | [Retry:Login:com.vmware.vim.eam:381bc5710df87f9f] Re-login to vCenter because method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null
YYYY-MM-DDTHH:MM:SS | INFO | vim-monitor | OpId.java | 37 | [vim:loginExtensionByCertificate:6c050a9ef5b82da8] created from [Retry:Login:com.vmware.vim.eam:381bc5710df87f9f]
YYYY-MM-DDTHH:MM:SS | INFO | vim-async-0 | OpIdLogger.java | 43 | [vim:loginExtensionByCertificate:6c050a9ef5b82da8] Failed.
YYYY-MM-DDTHH:MM:SS | WARN | vim-async-0 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:381bc5710df87f9f] Re-login failed, due to:
com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.
at com.vmware.eam.vim.security.impl.SessionManager.convertLoginException(SessionManager.java:295) ~[eam-server.jar:?]
at com.vmware.eam.vim.security.impl.SessionManager.lambda$loginExtension$4(SessionManager.java:154) ~[eam-server.jar:?]
at com.vmware.eam.async.remote.Completion.onError(Completion.java:86) [eam-server.jar:?]
at com.vmware.eam.vmomi.async.FutureAdapter.setException(FutureAdapter.java:81) [eam-server.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$ClientFutureAdapter.setException(MethodInvocationHandlerImpl.java:731) [vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.fail(MethodInvocationHandlerImpl.java:578) [vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture$RetryActionImpl.proceed(MethodInvocationHandlerImpl.java:625) [vlsi-client.jar:?]
at com.vmware.eam.vim.security.impl.ExtensionSessionRenewer.retry(ExtensionSessionRenewer.java:149) [eam-server.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.setException(MethodInvocationHandlerImpl.java:541) [vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:239) [vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:271) [vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54) [vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24) [vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:57) [vlsi-client.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_402]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_402]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_402]
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.
at sun.reflect.GeneratedConstructorAccessor57.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_402]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_402]
at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_402]
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:284) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:241) ~[vlsi-client.jar:?]
... 7 more
YYYY-MM-DDTHH:MM:SS | ERROR | vim-monitor | VcListener.java | 124 | An unexpected error in the changes polling loop
com.vmware.eam.EamRemoteSystemException: Unexpected error communicating with the vCenter server.
at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:106) ~[eam-server.jar:?]
at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78) ~[eam-server.jar:?]
at com.vmware.eam.vc.VcListener.main(VcListener.java:140) ~[eam-server.jar:?]
at com.vmware.eam.vc.VcListener.call(VcListener.java:118) [eam-server.jar:?]
at com.vmware.eam.vc.VcListener.call(VcListener.java:58) [eam-server.jar:?]
at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]
at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_402]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_402]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_402]
Caused by: com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.
YYYY-MM-DDTHH:MM:SS | INFO | vim-monitor | VcListener.java | 125 | Full stack trace: com.vmware.eam.EamRemoteSystemException: Unexpected error communicating with the vCenter server.
at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:106)
at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78)
at com.vmware.eam.vc.VcListener.main(VcListener.java:140)
at com.vmware.eam.vc.VcListener.call(VcListener.java:118)
at com.vmware.eam.vc.VcListener.call(VcListener.java:58)
at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58)
at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: (vim.fault.NotAuthenticated) {
faultCause = null,
faultMessage = null,
object = ManagedObjectReference: type = Folder, value = group-d1, serverGuid = 82401a9c-d8d4-410b-8da7-5f274d166fcf,
privilegeId = System.View
}
Environment
VMware vCenter Server
Cause
- Because in the VCSA EAM extension was missing from the vCenter the EAM was failing with Re-login failed.
- When you run to replace the Solution user certificate using vCert Script it did not show updated/matches certificate for eam extension thumbprint.
- Check from VCDB the vpx_ext table to validated that eam extension (com.vmware.vim.eam) was not present
psql -U postgres -d VCDB -c "SELECT * FROM vpx_ext;"
Resolution
Note: Take VCSA snapshot of the VCSA VM before making changes.
https://knowledge.broadcom.com/external/article/318333/snapshot-best-practices-for-vcenter-serv.html
- Connect to vCenter server via SSH with root access.
- Stop EAM service:
service-control --stop vmware-eam - Make sure the /etc/vmware-eam/firstboot/extension/extension.xml.installer file is correct:
-
The "config - extension - certificate" XML tag has the following contents:
<certificate>@CERT_PATH@</certificate> -
The "config - extension - health - url" XML tag has the following contents:
<url>https://@HOST@:443/eam/eamService-web/health.xml</url> -
The "config - extension - servers - server - url" XML tag has the following contents:
<url>https://@HOST@:443/eam/sdk/</url> -
The "config - extension - managedEntityInfos - managedEntityInfo - smallIconUrl" XML tag has the following contents:
<smallIconUrl>http://@HOST@:80/eam/imx/esx_vm_16x.png</smallIconUrl> -
The "config - extension - managedEntityInfos - managedEntityInfo - iconUrl" XML tag has the following contents:
<iconUrl>http://@HOST@:80/eam/imx/esx_vm_32x.png</iconUrl>
-
- Make a backup of the file /usr/lib/vmware-eam/firstboot/eam_firstboot.py:
cp /usr/lib/vmware-eam/firstboot/eam_firstboot.py /usr/lib/vmware-eam/firstboot/eam_firstboot.py.bak
- Manipulate the file /usr/lib/vmware-eam/firstboot/eam_firstboot.py
- Change the following lines -
if fbAction == FBActions.FIRSTBOOT:
fb.setup_config_files()
fb.register_with_cm()
fb.installService()
fb.registerExtension()
fb.setup_service_security()
upgrade_dir = fb.getUpgradeDir()
if(upgrade_dir):
fb.importDepotTrustJson(upgrade_dir)
startService(fb)
To look like -
if fbAction == FBActions.FIRSTBOOT:
#fb.setup_config_files()
#fb.register_with_cm()
#fb.installService()
fb.registerExtension() <<< DO NOT COMMENT THIS LINE
#fb.setup_service_security()
#upgrade_dir = fb.getUpgradeDir()
#if(upgrade_dir):
# fb.importDepotTrustJson(upgrade_dir)
#startService(fb) - Execute the following command:
PYTHONPATH=$PYTHONPATH:$VMWARE_PYTHON_PATH $VMWARE_PYTHON_BIN -B /usr/lib/vmware-eam/firstboot/eam_firstboot.py
- Check the vpx_ext table to validate if the eam extension (com.vmware.vim.eam) is present:
psql -U postgres -d VCDB -c "SELECT * FROM vpx_ext;"
- Ensure eam's service registration is present:
/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert | grep -i "Service Type:" | sort | uniq -c
- Revert the changes done to the eam_firstboot.py file.
- Start EAM service:
service-control --start vmware-eam
- Proceed with VCSA Pre-check and Stage -2 to upgrade the VCSA.
Additional Information
Feedback
