Error: "Could not connect to one or more vCenter Server Systems" in the vSphere Client due to duplicate endpoints

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

The VMware vSphere Client displays the below error:

Failed to verify the SSL certificate for one or more vCenter Server Systems: https://<vCenter-FQDN-Or-IP>:443/sdk
Could not connect to one or more vCenter Server Systems: https://<vCenter-FQDN-Or-IP>:443/sdk

This article provides information to remove the stale service registrations from vCenter server using SSH session.
After upgrading or reinstalling vCenter Server (notably to versions 7.x or 8.x), the lsdoctor tool may report duplicate service registrations within the SSO domain. This occurs when new endpoints are registered without the successful removal of previous ones.
When running python lsdoctor.py -l.(--lscheck ) the following error is reported:

ERROR generateReport: [FQDN] (VC 7.0 or CGW) found Duplicates Found: Ignore if this is the PSC HA VIP. Otherwise, you must unregister the extra endpoints.

When running python vdt.py (Using the VCF Diagnostic Tool for vSphere (VDT)), an error similar to the following is reported in the SSO Checks section:

* [FAIL] ######## (VC Server or CGW)

[FAIL] Duplicates Found
You must unregister the extra endpoints. Please run lsdoctor.py -r option 3 on this node for each service type listed

Duplicates by Node ID: 'Node ID: NO_NODE_ID'"
'Service type: <service1>': - 'Service ID: <serivce1 ID>'

Note the NO_NODE_ID value for Node ID.

vCenter server reports error: "Possible duplicate entries in the Lookup Service Registration Database detected"
Manual method to remove stale service registrations from vCenter Server
Steps in this KB article is applicable for the duplicate service registrations with the same Service Type, Node ID, Owner ID, Site ID and version

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

This issue occurs in these situations when using Enhanced Linked Mode:

Improper decommissioning of external PSC or vCenter in Enhanced Linked Mode (ELM).
During upgrade or migration, service and certificate records are coupled with the vCenter Machine ID. If a mismatch occurs, the Lookup Service Registration Database may retain stale entries for services.
Another vCenter Server in the Single Sign-On (SSO) domain has restarted or is not fully available after restart.
During the re-installation of the vCenter Server, it is possible to have the same vCenter Server registered more than once to the Single Sign-On (SSO).
With a previous install of vCenter Server, SSL certificates were not overwritten or appropriately removed during an upgrade or re-installation.

Resolution

Important: Contact Broadcom support to obtain assistance to safely proceed with below solution steps.

Note:

Capture offline snapshots for vCenters in linked mode.
This issue may be transient as another vCenter Server in an Enhanced Link Mode domain is restarting. Before continuing with troubleshooting, it is advised to wait 10 minutes, log out, and log back into the vCenter Server. The error may clear on its own. In addition, it is strongly advised to determine if logging into the other vCenter Server identified in the error message directly to determine if vCenter services are up and running before continuing. If all vCenter Servers are up and running and this error persists, continue with this resolution to identify duplicate service registrations or other errors.

Steps to check stale registrations on vCenter Server:

1. List of the services registered within the vCenter:

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert > /tmp/psc.txt

2. Validate the number of service registrations for each vCenter:

less /tmp/psc.txt | grep -i "Service Type: " | sort | uniq -c
For an environment with two vCenters in Enhanced Linked Mode (ELM), there would be two service registrations for each service. If there are more than two, then these can be potentially stale ones.

3. Identify a valid registration against stale registration using /etc/vmware/install-defaults/vmdir.ldu-guid to compare Node ID from above file (/tmp/psc.txt):

For Example:
- The below service registration belongs to the mentioned Node ID.
  Name: ServiceName
  Description: ServiceDescription
  Service Product: com.vmware.cis
  Service Type: ServiceType
  Service ID: 1f10XX8f-438d-XXb-90f8-47e7d6XXa94f
  Site ID: default-first-site
  Node ID: b2b7XX1e-5XX9-4a55-88XX-699fXXeb1d6f
  Owner ID: vpxd-extension@vsphere.local
  Version: 1.0
  Endpoints:
  Type: com.vmware.cis.data.provider
  Protocol: vmomi
  URL: https://vcsa.example.org:443/sdk
  SSL trust:
  Endpoint Attributes:
  cis.common.ep.localurl: http://localhost:11080/sdk
  Attributes:
  com.vmware.cis.cm.HostId: 603b3XX1-fXXb-4XX2-aXX5-81517XX8f47a
- The Node ID should match the entry in /etc/vmware/install-defaults/vmdir.ldu-guid.

Scenario 1: When the Node ID is same between service registrations which needs to be removed.

For Example:
Below 2 service registration have the same Node ID, Service Type, Owner ID.

    Name: ServiceName
    Description: ServiceDescription
    Service Product: com.vmware.cis
    Service Type: ServiceType
    Service ID: 1f10XX8f-438d-XXb-90f8-XXXd6XXa94f
    Site ID: default-first-site
    Node ID: b2b7XX1e-5XX9-4a55-88XX-699fXXeb1d6f
    Owner ID: vpxd-extension@sso_domain

Name: ServiceName
Description: ServiceDescription
Service Product: com.vmware.cis
Service Type: ServiceType
Service ID: 789XXXXfg-64e3-XXb-32u9-XXX862XXX
Site ID: default-first-site
Node ID: b2b7XX1e-5XX9-4a55-88XX-699fXXeb1d6f
Owner ID: vpxd-extension@sso_domain

In this case we need to identify the stale entry to be removed using the vCenter server logs.

Firsboot log will have all the registered service registration entries which are downloaded during the vCenter server installation, from here we can get the valid service ID which was installed in vCenter server.
Go to the location /var/log/firstboot in the vCenter server logs and run the command grep -i "Service ID" * | less (service ID for which the duplicate entry is detected)
If an output as below is seen for the service ID, which means that service ID is currently installed in the vCenter server and it is a valid one which should not be removed.

- vmidentity-firstboot.py_2##3_stdout.log:XXXX-0x-XXTXX:XX:XX.XXXZ Running command: ['/usr/lib/vmware-lookupsvc/tools/lstool.py', 'register', '--id', 'Service ID', '--user', 'Administrator@sso_domain', '--password', '*******', '--spec', '/var/lib/lookupsvc/lookupsvc syaas.service.spec', '--url', 'https://vCenter server FQDN.au:443/lookupservice/sdk']

Following steps need to be performed to remove the stale entries identified above:

Take a no memory snapshot of the vCenter server if it is standalone or powered off snapshots of vCenter servers if in linked mode
Take SSH access to the vCenter server and run the following command to unregister the stale service ID:

- /usr/lib/vmware-lookupsvc/tools/lstool.py unregister --url http://localhost:7090/lookupservice/sdk --id Stale_Service_ID --user 'administrator@sso_domain' --password 'sso_password' --no-check-cert

Scenario 2: When the Node ID is different between duplicate service registrations which needs to be removed

Refer the following KB to remove stale entry using a script when the Node ID's are different:

Script to remove stale service registrations from vCenter Server.

Scenario 3: When the Node ID is not present to validate the duplicate service registrations. Also, we do not see any firstboot logs related to the specific service.

Execute the command to list the service registrations on the vCenter.

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert > /tmp/psc.txt

Check for the number of endpoints for the specific service registration.
Identify which service registration has the least number of endpoints. That specific service ID can be removed as duplicate.

Note: Ensure that a proper snapshot is taken for the vCenter before making any changes.

Additional Information

Using the VCF Diagnostic Tool for vSphere (VDT)

Using the 'lsdoctor' Tool