Upgrade to vCenter Server 8.0.x fails with "ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Upgrade to vCenter Server 8.0.x fails with the following error during pre-check:

  Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted
  certificates by the System!
  
  Verify that the ESX Agent Manager extension is running properly on the source vCenter Server instance
  and https://VC_IP/eam/mob presents correct data. If log in to the MOB is not successful, try resolving
  the issue with https://kb.vmware.com/s/article/94934.

• for example:

• when looking in /var/log/vmware/eam/eam.log in the source vCenter Server, the following errors can be found:

  eam.log
  ..
  2023-12-19T04:19:59.764Z |  INFO | vim-monitor | VcListener.java | 125 | Full stack trace: com.vmware.eam.EamRemoteSystemException: Client error communicating with the > vCenter server.
  at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:103)
  at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78)
  ..
  Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
  at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:56)
  at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:271)
  at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54)
  at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24)
  at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:57)
  at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:227)
  at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)
  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)
  at com.sun.proxy.$Proxy50.currentTime(Unknown Source)
  at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:101)
  ... 9 more
  ..

 

• When querying the PNID on the source vCenter, you find it to be in small letters:

  # /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
  vcsa.domain.local

• However, when looking in /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json in the source vCenter, its hostname portion is spelled in upper-case:

  VCSA.domain.local

Environment

VMware vCenter Server 7.0.x

Cause

This issue occurs, when there is a difference in the case-sensitivity between the vCenters FQDN, its primary network identifier (PNID), the domain controller name (DCName) of its embedded platform service controller in the vsphere.local domain, and the spelling of the PNID in /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json in vCenter Server 7.0.

In vCenter Server 8.x, /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json no longer contains the vCenter FQDN, but replaces it with localhost.

Resolution

Caution: Before applying the steps below, please take a backup or an offline-snapshot (in powered-off state) of the vCenter Server Appliance . If the vCenter is part of a Linked Mode replication setup, please backup/snapshot all replicating nodes as well.

To resolve this issue, change the PNID of the vCenter Server to correspond with the spelling in /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json. However, since you cannot do a direct change to the same PNID in a different case, this is a 2-step process:

1. Create a temporary record in your DNS server with the current vCenter IP and a temporary FQDN (e.g. VCSA1.domain.local)
2. Change the PNID to this temporary FQDN following the complete process outlined in https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-F46DBE63-F04E-42A1-B940-63A8F5B86ACF.html
3. Do a second PNID change back to the old FQDN, but using the spelling that was found in /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json. For example, if the vCenter FQDN is listed as VCSA.mj.local in ProxyConfiguration.json, use this as the new PNID
4. remove the temporary DNS record
5. Upgrade the vCenter Server