vCenter Server UI loads slowly with connection errors after upgrading to version 9.1
search cancel

vCenter Server UI loads slowly with connection errors after upgrading to version 9.1

book

Article ID: 444002

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

After upgrading the vCenter Server to version 9.1, users experience slow UI performance. A banner message is displayed: Could not connect to one or more vCenter Server systems: https://<FQDN_OF_STALE_vCenterServer>:443/sdk

Additional observations:

  • Replication status shows the vCenter Server is not replicating to any other vCenter.
  • The vCenter Server was previously part of VMware Cloud (VMC) on AWS and was decommissioned from cloud to on-prem.

The following errors are found in the logs:

/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

YYY-MM-SSTHH:MM:SS ERROR vsphere-ui 215 [vc@4413 threadName="wcp-plugins-notifier-1" logger="c.v.v.plugin.notification.WcpServicePluginNotificationsListener" ] Unable to subscribe to vc <FQDN_OF_STALE_vCenterServer> (9a9###c0-f##f-4##9-b##f-6e######35adce), nodeId: dc###c84-1##0-4##d-8##c-d9####70373 java.util.concurrent.CompletionException: com.vmware.vcenter.apigw.api.sso.SsoServiceException: Failed to acquire a token by token
...
Caused by: com.vmware.vcenter.apigw.api.sso.SsoServiceException: Failed to retrieve an STS client for SSO domain <Stale_Domain>.local(0####09-e##5-4##f-b##0-5d##9c7~CLOUD)

/var/log/vmware/vsphere-ui/logs/apigw.log

YYY-MM-SSTHH:MM:SS ERROR vsphere-ui 8546 [vc@4413 threadName="agw-token-####" logger="com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl" SsoServiceImpl ] Failed to build an STS client for domain <Stale_Domain>.local(0####09-e##5-4##f-b##0-5d##9c7~CLOUD) in 30021 ms com.vmware.vcenter.apigw.api.sso.SsoServiceException: Failed to build a SecurityTokenServiceConfig
...
Caused by: com.vmware.vcenter.apigw.api.sso.SsoServiceException: Failed to retrieve the STS issuers certificates for domain <Stale_Domain>.local(0####09-e##5-4##f-b##0-5d##9c7~CLOUD) in 30021 ms. (URL: http://localhost:1080/external-vecs/http2/<FQDN_OF_STALE_vCenterServer>/443/sso-adminserver/sdk/<Stale_Domain>)

/var/log/vmware/vpxd/vpxd.log

YYY-MM-SSTHH:MM:SS ERROR vpxd 5528 [vc@4413 sub="cnxmgrLogger" opID="RemoteInstancePreload-75####8c"] Failed to preload instance STS cert; instId: 9#####c0-f##f-4##9-b##f-6e025435adce, host: <FQDN_OF_STALE_vCenterServer> ex: N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)

/var/log/vmware/vgfa/ls_configurator/ls_configurator.log shows the stale VMC vCenter is still visible:

YYY-MM-SSTHH:MM:SS INFO VGFA 40519 [vc@4413] [CONF_Manager]++++++++++++++ ADDED VC: <FQDN_OF_STALE_vCenterServer>

 

Environment

VCF 9.1

Cause

This issue is caused by a stale registration of a decommissioned VMware Cloud (VMC) on AWS vCenter within the on-premises Lookup Service. Although the Enhanced Linked Mode (ELM) relationship was broken or the cloud SDDC deleted, the registration was not properly cleaned up.

On-premises services (VGFA and API Gateway) discover this stale endpoint during startup. When a user logs in, the system attempts to connect to the stale vCenter SDK and SSO endpoints. This triggers multiple 15- to 30-second timeouts, causing cumulative login delays of 75 seconds to 5 minutes.

Resolution

To resolve this issue, delete the stale registration using one of the following methods.

Take a snapshot of the vCenter Server and ensure a valid file-level backup exists.

Method 1: Using JXplorer (UI)

  1. Download and install JXplorer from jxplorer.

  2. Connect to vSphere Single Sign-On following the steps in Using JXplorer to connect to the vSphere Single Sign-on

  3. Navigate to: World > local > vsphere > configuration > ForeignDomains.

  4. Delete the contents of the ForeignDomains folder containing the stale vCenter registrations.

Method 2: Using Command Line (LDAP)

If JXplorer cannot be used, run the following commands from the vCenter Server appliance:

  1. Search for the stale registration:

    /opt/likewise/bin/ldapsearch -H ldap://localhost:389 -x -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W -s one -b "cn=ForeignDomains,cn=Configuration,dc=vsphere,dc=local"

  2. Delete the stale entry (Replace the cn value with the ID found in step 1):

    /opt/likewise/bin/ldapdelete -H ldap://localhost:389 -x -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w '<SSO_PASSWORD>' -r "cn=5###d5c-c##a-4##5-b##9-3d#####3ff5,cn=ForeignDomains,cn=Configuration,dc=vsphere,dc=local"
Powered by Wolken Software