"Failed to connect to lookup service..." error when configuring the lookup service on the Cloud Director Availability appliances

Products

VMware Cloud Director VMware vCenter Server

Issue/Introduction

Symptoms:

During the initial configuration of a Cloud Director Availability appliance, configuring the lookup service fails and you see a similar error:

Failed to connect to lookup service at https://PSC_Address:443/lookupservice/sdk

In /opt/vmware/h4/<cloud|manager|replicator|tunnel>/log/<cloud|manager|replicator|tunnel>.log on the Cloud Director Availability appliance, you see a similar error:

2020-04-27 11:51:14.470 ERROR - [UI__55ef4eba-a6c7-444a-9fcb-1049fe259f2a_gh] [https-jsse-nio-8440-exec-3] c.v.h4.common.service.BaseConfigService : Failed to connect to lookup service at https://PSC_Address:443/lookupservice/sdk.

com.vmware.exception.GenericSSLException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
    at com.vmware.exception.converter.ClientExceptionConverter.convertException(ClientExceptionConverter.java:64)
    at com.vmware.vlsi.util.ExceptionConverterInterceptor.handleException(ExceptionConverterInterceptor.java:30)
    at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:254)
    ...
Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: SSL handshake from 0.0.0.0/0.0.0.0:59746 to PSC_Address:443 failed in 113 ms
    at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.handleHandshakeException(ThumbprintTrustManager.java:597)
    at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.verify(ThumbprintTrustManager.java:422)
    at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.verifyHostname(VlsiSslSocketFactory.java:129)
    at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.createLayeredSocket(VlsiSslSocketFactory.java:122)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCloud Availability 3.0.x
VMware Cloud Director Availability 4.x
VMware vCloud Availability 3.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server 7.0.x

Cause

This issue can occur when:

The machine SSL certificate of a vCenter Server certificate differs from the certificate stored in the lookup service.
The lookup service certificate on port 443 differs to the certificate on port 7444.

Resolution

To verify you are encountering vCenter/lookup service certificate issues, perform the following checks:

Lookup service certificate mismatch

SSH to the Cloud Director Availability On-Premises appliance and log in as root.
Run the following commands against the Platform Services Controller:

openssl s_client -connect PSC_Address:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout
openssl s_client -connect PSC_Address:7444 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout

Compare the outputs to determine if there is a mismatch.

vCenter Server certificate mismatch

SSH to the Platform Services Controller and log in as root.
Use the lstool script to get a list of the registered services on the PSC:

vSphere 6.x:
/usr/lib/vmidentity/tools/scripts/lstool.py list --url http://localhost:7080/lookupservice/sdk > /tmp/services.txt

vSphere 7.0:
/usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk > /tmp/services.txt

Open the services.txt file and search for the following section:

Service Type: vcenterserver

Take note of the endpoint certificate for the service.
Run the following command against the vCenter Server:

openssl s_client -connect vCenter_Address:443 < /dev/null 2>/dev/null | openssl x509

Compare the certificates from steps 4 and 5 to determine if there is a mismatch.

To resolve any certificate mismatch issues, contact VMware Support and note this Article ID (78920) in the problem description. For more information, see How to Submit a Support Request.

Additional Information

For more information on vSphere certificate issues, see:

Impact/Risks:
Warning: Incorrectly updating certificate information of service registrations may break the functionality of that service.