Site pairing for vSphere Replication fails with error: Unable to login to HBR Management Server at port 8043

Products

• VMware Live Recovery
• VMware vCenter Server

Issue/Introduction

Symptoms:

• In the Site Recovery UI, the following error is displayed when attempting to establish a site pair:

"Unable to retrieve pairs from extension server at https://hostname:8043. Unable to login to 'HBR Management Server at https://hostname:8043'"

• The issue occurs if the vCenter Server is upgraded or the MACHINE_SSL_CERT certificate is renewed.

• Reconfiguration of the vSphere Replication appliance fails.

Environment

• VMware vSphere Replication 8.x
• VMware vSphere Replication 9.x
• vCenter Server 7.x
• vCenter Server 8.x

Cause

• A thumbprint mismatch will be identified on the vCenter Server.
• This issue occurs when the SSL certificate thumbprint registered in the Lookup Service differs from the thumbprint of the SSL certificate currently presented by the vCenter Server. This mismatch prevents successful authentication during the site pairing process.
• Log review of /opt/vmware/support/logs/dr-client/dr.log from the vSphere Replication appliance shows errors indicating a failure to connect to the Lookup Service due to a certificate thumbprint mismatch:

ERROR com.vmware.vr.client.replications.VrSiteIssuesDataHandler ... getPairSrmSummaryIssues - Cannot retrieve vSphere Replication site issues.
com.vmware.srm.client.topology.client.vmomi.Service$LoginFailedException: Unable to login to 'HBR Management Server at https://hostname:8043'.
...
Caused by: (hms.fault.CannotVerifyCredentialsFault) {
   faultCause = (hms.fault.HmsRuntimeFault) {
      faultCause = (hms.fault.HmsRuntimeFault) {
         faultCause = (hms.fault.HmsRuntimeFault) {
            faultCause = null,
            faultMessage = null,
            originalMessage = Thumbprint mismatch
         },
         faultMessage = null,
         originalMessage = com.vmware.vim.vmomi.core.exception.CertificateValidationException: Thumbprint mismatch
      },
      faultMessage = null,
      originalMessage = com.vmware.vim.vmomi.core.exception.CertificateValidationException: Thumbprint mismatch

[srm-reactive-thread-5] INFO  com.vmware.srm.client.topology.impl.vmomi.vlsi.BackOffRetryWrapper  510fd54f-6092-####-####-4090bd95cda6  - Invoking retryable op for 'RetrieveContent' for Lookup Service at hostname#695160674
[srm-reactive-thread-8] WARN  com.vmware.srm.client.infrastructure.init.workflow.Configurator  510fd54f-6092-####-####-4090bd95cda6  - Failed to create websso context:
com.vmware.vim.vmomi.client.exception.SslException: Unable to connect to Lookup Service at https://hostname:443/lookupservice/sdk. Reason: javax.net.ssl.SSLException: Certificate thumbprint mismatch.

• Logs from /opt/vmware/hms/logs/hms.log further confirm the certificate mismatch:

Caused by: com.vmware.vim.binding.hms.fault.HmsRuntimeFault: javax.net.ssl.SSLException: Certificate thumbprint mismatch, expected: 1E:EE:25:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:BC:30:A7:53 but encountered:2B:4C:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:A4 or 1D:EE:80:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:97:55:13:C7

• To confirm, execute lsdoctor on the vCenter Server, which also reports a thumbprint mismatch between the Lookup Service and the active vCenter certificate.

# python lsdoctor.py -l

yyyy-mm-ddThh:mm:ss INFO main: You are reporting on problems found across the SSO domain in the lookup service.  This doesn't make changes.
yyyy-mm-ddThh:mm:ss INFO live_checkCerts: Checking services for trust mismatches...
yyyy-mm-ddThh:mm:ss INFO generateReport: Listing lookup service problems found in SSO domain
yyyy-mm-ddThh:mm:ss INFO generateReport: No issues detected in the lookup service entries for hostname  (vSphere Replication).
yyyy-mm-ddThh:mm:ss ERROR generateReport: default-site\hostname  (SRM) found SSL Trust Mismatch: Please run python ls_doctor.py --trustfix option on this node.
yyyy-mm-ddThh:mm:ss INFO generateReport: No issues detected in the lookup service entries for ##NO_HOSTNAME##.
yyyy-mm-ddThh:mm:ss INFO generateReport: Report generated:  /var/log/vmware/lsdoctor/hostname -yyyy-mm-dd-075444.json

Resolution

• Execute the lsdoctor utility on the vCenter Server to identify and resolve the SSL thumbprint mismatch.

• Use the command below to perform automatic remediation. This option corrects SSL trust mismatch issues in the lookup service.

python ls_doctor.py --trustfix

• Follow the tool's guidance to update the registered certificate thumbprint in the Lookup Service. Using the 'lsdoctor' Tool
• Reconfigure the vSphere replication appliance once the trust mismatch issue is fixed.

 

 

 

Additional Information

Site Recovery Manager & vSphere Replication Appliance disconnects from vCenter when Machine SSL certificate is replaced