SSL Certificate Thumbprint Mismatch Error in Site Recovery Manager after vCenter Certificate replacement

Products

VMware Live Recovery VMware vCenter Server

Issue/Introduction

SRM (Site Recovery Manager) tries to connect to old thumbprints of vCenter server or SRM and it fails with thumbprint mismatch error.
SRM server shows "Not connected" on Site Recovery page, "Reconnect" from Site Recovery UI may fail or be successful but status still shows "Not Connected".

"SRM Server cannot connect to SRM Server at 'https://####:443/drserver/vcdr/vmomi/sdk". SRM server #### cannot validate SSL certificate from server at ####:443. The remote host certificate has these problems: Unknown SSL certificate error."
Another way the Site Recovery - Site Pair page may look like:
The local SRM shows Remote SRM connection as "Not connected" and the target SRM shows "Unknown". The target SRM server also shows as "unknown:9086".
The below error message may also be seen on Site Recovery page:

"Unable to connect to Lookup Service at https://####:443/lookupservice/sdk. Reason: javax.net.ssl.SSLException: Certificate thumbprint mismatch."

Another error may appear once you hit Site Recovery from VC:

"...CertificateValidationException: server certificate is not trusted and thumbprint verification is not configured"

Environment

VMware Site Recovery Manager 8.x

VMware Live Site Recovery 9.x

Cause

Incorrect thumbprints on SRM Database tables is the cause of this issue.
A sync issue may occur if the SRM hostname differs from the real name listed in the certificate. This preserves outdated entries and does not update thumbprints.
Thumbprints may fail if SRM is upgraded at any point while vCenter is out of sync.
Thumbprint mismatch issues will arise if a certificate is renewed or changed on the vCenter server or the SRM/VR server. Both SRM and VR servers must undergo immediate reconfiguration in order for the vCenter certifications to be updated.

In the SRM /opt/vmware/support/logs/srm/vmware-dr.log the below entries can be seen:

"SSL Exception: Verification parameters:
--> PeerThumbprint: ##:##:##:##:##:##:##
--> ExpectedThumbprint: ##:##:##:##:##:##:##
--> The remote host certificate has these problems:
-->
--> * unable to get local issuer certificate"

The log entry above indicates that the database contains stale thumbprints, as it reports a different "PeerThumbprint" and "ExpectedThumbprint".
In the log, /opt/vmware/support/logs/dr-client/dr.log

####-##-## 05:57:38,916 [srm-reactive-thread-2973] WARN com.vmware.srm.client.infrastructure.http.BaseAsyncController #########-####-####-####-########## - Request for path 'login' failed.com.vmware.vim.vmomi.client.exception.SslException: Unable to connect to Lookup Service at https://#########.##########.###:443/lookupservice/sdk. Reason: javax.net.ssl.SSLException: Certificate thumbprint mismatch.
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:265)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:362)
at com.vmware.dr.ui.tools.utilities.ExecutorUtils.lambda$wrap$1(ExecutorUtils.java:36)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLException: Certificate thumbprint mismatch.
Because of stale entries, the log events indicate that the vCenter server's certificate thumbprint does not match.

Resolution

SRM/VR must be reconfigured in order for the thumbprints of newly or upgraded vcenter servers to be updated in SRM/VR databases.

Note: Please capture SRM, VR and VC Snapshots before performing appliance reconfiguration.

Refer to the document to reconfigure SRM - Reconfigure the Site Recovery Manager Appliance.

Refer the document to reconfigure vSphere Replication appliance - Reconfigure vSphere Replication

Perform reconnect of Site Pair post reconfiguring SRM's, refer - Reconnect the Connection Between Sites.

Another approach, if above steps fail to resolve the thumbprint mismatch issue, use the lsdoctor script as follows:

Download the lsdoctor script from KB-80469
Upload the script to the vCenter Server.
Navigate to the directory where the script was uploaded.
Extract the contents of the ZIP file:
- unzip lsdoctor.zip
Run the below command for issue check:
- python lsdoctor.py --l
Run the following command to fix the thumbprint mismatch (if reported):
- python lsdoctor.py --trustfix
After executing the script, restart the vCenter services to complete the fix.
- service-control --stop --all && service-control --start --all

If the issue persists after running the script, or if the symptoms and issue described above match your scenario but the resolution steps do not help, please contact Broadcom Support for further investigation and assistance.