Enhanced Replication Mappings Error: Fault occurred while performing health check. Details: 'Connect: certificate verify failed (SSL routines)'.

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• Enhanced Replication mappings show the following error when running a 'RUN TEST' action:
  • Fault occurred while performing health check. Details: 'Connect: certificate verify failed (SSL routines)'.
  • 
    
    
• Enhanced Replication mappings succeed from one direction (vCenter1 > vCenter2), but from from the other direction (vCenter2 > vCenter1)
  
  
• You see see the following replication error:
  • 'A replication error occurred at the vSphere Replication Server for replication '<VM-Name>'. Details 'No connection to VR Server for virtual machine <VM-Name> on host <ESXi-Hostname> in cluster <Cluster-Name>: Unknown'.
    
    
  • 

Environment

vSphere Replication 9.x

Cause

We see this type of behavior after a vSphere Replication certificate renewal.

As a result, one of the vSphere Replication appliances has a stale certificate and thumbprint entry in it's 'hbrserverentity' datastore.

This stale certificate and thumbprint entry will cause communications issues between the paired vSphere Replication appliance and the ESXi hosts on the paired site.

Resolution

Review the certificate being presented on the local side vSphere Replications hms.log for the paired vSphere Replication appliance.

Review this certificate against the output of the 'openssl' command ran on the paired vSphere Replication appliance.

If these certificates do not match, open up a support request with Broadcom Support for further assistance.

Additional Information

Reviewing the certificate in the the hms log file:

1. Open an SSH session to the vSphere Replication appliance that has the failing enhanced replication mappings errors
2. Log in with admin
   1. Run 'sudo -i' to login to the root account (this will ask for the admin password)
3. Navigate to the following directory
   1. cd /opt/vmware/hms/logs
4. Run the following command to review the hms log file:
   1. cat hms.log | less
5. Once you have hms.log pulled up, enter a '/' to enable the search function
6. We will search for the paired vSphere Replication IP address using the following query (must provide your IP address):
   1. 'ipAddress = xx.xx.xx.xx'
   2. The output will look similar to this:
      1. 

=====================

Run the following command on the vSphere Replication appliance that does not have the enhanced replication mappings errors:

echo | openssl s_client -connect xx.xx.xx.xx:443

*Note: The IP address used in the command above needs to match the IP address you searched for in the hms.log.

=====================

Review both certificates to ensure that they match. You can review either the beginning or ending section of the certificate.