Unable to see other Linked-Mode vCenter Servers in inventory

Products

VMware vCenter Server

Issue/Introduction

There are two or more vCenter Servers in a Linked-Mode
When logged into the Web Client for vCenterA, you see vCenterA and vCenterB in the inventory, but when logged into vCenterB, you only see vCenterB in the inventory and vCenterA is not shown.
In the vCenter - /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log file, you see entries similar to:

[<YYYY-MM-DD>T<TIME>] [INFO ] aggregator-query-service-152 70000066 100004 200001 com.vmware.vise.search.transport.impl.AggregatorQueryServiceImpl QueryService LinkedQueryService(https://vcenterfqdn:443/invsvc) failed to respond: com.vmware.vise.search.transport.HostConnectException: Unable to connect to VMware Inventory Service -(https://vcenterfqdn:443/invsvc). 
com.vmware.vise.search.transport.HostConnectException: Unable to connect to VMware Inventory Service - (https://vcenterfqdn:443/invsvc)
    at com.vmware.vise.util.concurrent.ExecutorUtil$2.run(ExecutorUtil.java:195)
    at com.vmware.vise.util.concurrent.ExecutorUtil$ThreadContextPropagatingRunnable.run(ExecutorUtil.java:928)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified
    at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:251)

Cause

This issue occurs when one of the vCenter Servers in the Linked-Mode environment cannot verify the certificate chain of another vCenter Server. Due to this the connection fails and as a result the other vCenter Servers are not shown in the inventory.

Resolution

We need to review the certificates in the VMware Endpoint Certificate Store (VECS) to verify that the signing certificates of the other Linked-Mode vCenter Server nodes are present. If any are missing, we will import them

To resolve the issue, follow the steps below:

SSH into the node which cannot see one or more of the Linked-Mode vCenter Servers nodes.
Run the command below to get the output of the TRUSTED_ROOTS certificate store in the VECS:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep -E "Subject:|Issuer:|Not Before:|Not After|Alias"

You should see output similar to:

Alias : 9e#######c89d0c5c7d0aa####52c19e7913
        Issuer: CN=<FDQN of VCSA2>, DC=vsphere, DC=local, C=US, ST=California, O=<FDQN of VCSA2>, OU=VMware Engineering
            Not Before: <MM DD hh:mm:ss YYYY> GMT
            Not After : <MM DD hh:mm:ss YYYY> GMT
        Subject: CN=<FDQN of VCSA2>, DC=vsphere, DC=local, C=US, ST=California, O=<FDQN of VCSA2>, OU=VMware Engineering

Alias : 5######8ffdd3d508652855b9######0e0c51d19
        Issuer: CN=<FDQN of VCSA1>, DC=vsphere, DC=local, C=US, ST=California, O=<FDQN of VCSA1>, OU=VMware Engineering
            Not Before: <MM DD hh:mm:ss YYYY> GMT
            Not After : <MM DD hh:mm:ss YYYY> GMT
        Subject: CN=<FDQN of VCSA1>, DC=vsphere, DC=local, C=US, ST=California, O=<FDQN of VCSA1>, OU=VMware Engineering

Note: If the Issuer and Subject are the same, this indicates that it is a signing or root certificate.

Review the CN value of the Issuer string for each signing certificate and check whether one signing certificate is present for each of the other nodes.
Next, connect to each of the nodes for which the cert is missing and export the signing certificate from the TRUSTED_ROOTS store.

For more information on exporting certificates, see Manually reviewing certificates in VMware Endpoint Certificate Store for vSphere 6.x and 7.x

From any node, import the missing signing certs into the VMware Directory Service (VMDIR) using the command below:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert <PATH_TO_CERTIFICATE> --login <VSPHERE_ADMINISTRATOR>

Example

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /tmp/certs/vcsa2_root.crt --login [email protected]

Next, force a push of all certificates in VMDIR to the VECS on each node in the Linked-Mode environment using the command:

/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
Log out of the Web Client and back in to confirm that all nodes of the Linked-Mode are now showing in the vCenter Server inventory again.

Additional Information

For more information on dir-cli and vecs-cli, see dir-cli Command Reference and vecs-cli Command Reference.

"Could not connect to one or more vCenter Server Systems: https://vCenterFQDN: 443/sdk" error in the vSphere Web Client