vCenter upgrade fails at stage 2 with error "Source vSphere ESX Agent Manager(EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System"

Products

• VMware vCenter Server

Issue/Introduction

vCenter upgrade fails with the following error, even with the eam extension present in the mob page, and the certificate is updated from the KB:318255

In the eam.log, you find a communication error "com.vmware.eam.EamRemoteSystemException: Client error communicating with the vCenter server."

Loga location: /var/log/vmware/eam/eam.log

DATE-##-##T##:##:##| ERROR | vlsi | DispatcherImpl.java | 450 | Internal server error during dispatch

com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.

        at com.vmware.eam.vmomi.EAMInitRequestFilter.handleBody(EAMInitRequestFilter.java:57) ~[eam-server.jar:?]

        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.handleBody(DispatcherImpl.java:355) [vlsi-server.jar:?]

        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.dispatch(DispatcherImpl.java:285) [vlsi-server.jar:?]

        at com.vmware.vim.vmomi.server.impl.DispatcherImpl.dispatch(DispatcherImpl.java:243) [vlsi-server.jar:?]

        at com.vmware.vim.vmomi.server.http.impl.CorrelationDispatcherTask.run(CorrelationDispatcherTask.java:58) [vlsi-server.jar:?]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]

Date-##-##T##:##:## | ERROR | vim-monitor | VcListener.java | 123 | An unexpected error in the changes polling loop

com.vmware.eam.EamRemoteSystemException: Client error communicating with the vCenter server.

        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:103) ~[eam-server.jar:?]

        at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78) ~[eam-server.jar:?]

        at com.vmware.eam.vc.VcListener.main(VcListener.java:139) ~[eam-server.jar:?]

        at com.vmware.eam.vc.VcListener.call(VcListener.java:117) [eam-server.jar:?]

        at com.vmware.eam.vc.VcListener.call(VcListener.java:57) [eam-server.jar:?]

        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]

        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]

Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404

Cause

FQDN URL is not correctly updated in the rhttpproxy-endpoint.conf file

Resolution

From the file: /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd-rhttpproxy-endpoint.conf

Make sure you have the correct vCenter PNID/fqdn in the URL:

/etc/vmware-rhttpproxy/endpoints.conf.d ]# cat vpxd-rhttpproxy-endpoint.conf

/sdk local 8085 redirect allow

/sdkTunnel localtunnel 8089 allow reject

sdkTunnel:8089 localtunnel 8089 allow reject

#.#.#.#:8089 localtunnel 8089 allow reject

/sdk-vpxd local 8085 reject allow

/mob namedpipe /var/run/vmware/vpxd-mob-pipe redirect allow

/vod namedpipe /var/run/vmware/vpxd-webserver-pipe redirect allow

/client/clients.xml namedpipe /var/run/vmware/vpxd-webserver-pipe allow allow

/nfc namedpipe /var/run/vmware/vpxd-webserver-pipe allow allow

/screen namedpipe /var/run/vmware/vpxd-webserver-pipe redirect allow

/qs namedpipe /var/run/vmware/vpxd-qsadapter-pipe redirect allow

/vcenter local 8093 redirect allow

/catalog namedpipe /var/run/vmware/vpxd-webserver-pipe redirect allow

For Example, the URL should be as follows:

vCenterFQDN:8089 localtunnel 8089 allow reject

Update the file and restart vmware-rhttpproxy and eam services:

# service-control --restart vmware-rhttpproxy
# service-control --restart vmware-eam

Note: IP address instead of FQDN will not work; make sure the hostname and PNID are the same, and update the vCenter PNID in this file