Thumbprint mismatch when configuring the VLSR appliance after changing _MACHINE_CERT certificate on vCenter.

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• The VLSR Reconfigure task fails citing a thumbprint mismatch.
  
  Reconfigure reports following error:
  
  
  
  

• Issue may be also be reported when vCenter FQDN or IP is changed which eventually needs Certificate replacement.

• The srm-server service is stopped on the VLSR appliance.

Environment

• VMware Live Site Recovery 8.x
  
  
• VMware Live Site Recovery 9.x

Cause

• The _MACHINE_CERT certificate on vCenter has been updated and the new vCenter certificate thumbprint has not been correctly propagated to the VLSR appliance.

• Due to this the VLSR can no longer authenticate with the vCenter.

• The VLSR /opt/vmware/support/logs/srm/vmware-dr.log throws below error:

  YYYY-MM-DDTHH:MM:SS.SSS +08:00 warning vmware-dr[01374] [SRM@6876 sub-Default connID=1kp-####] StubExcTranslator : Error while calling stub for 'lookup.ServiceInstance:ServiceInstance'
--> N7Vmacore3Ssl18SSLVerifyExceptionE SSL Exception: Verification parameters:
--> PeerThumbprint: #:#:#:#:#:#:#:#:#:#:#:#: #:#:#:#:#:#:#:#:#:#:#:#: #:#:#:#:#:#:#:#:#:#:#:#:#                               
--> ExpectedThumbprint:  #:#:#:#:#:#:#:#:#:#:#:#: #:#:#:#:#:#:#:#:#:#:#:#: #:#:#:#:#:#:#:#:#:#:#:#:#                       
--> ExpectedPeerName: {vcenter fqdn}
  
  Where:
  PeerThumbprint --> Is the new vCenter Thumbprint
  ExpectedThumbprint --> Is the previous vCenter Thumbprint

• This can happen if the VLSR appliance is not reconfigured after a change of the vCenter certificate.

Resolution

Steps to resolve this issue:

1. Identify the new vCenter certificate thumbprint. Replace the 'vc-fqdn' in this command and run on VLSR SSH session.

   #  echo | openssl s_client -connect vc-fqdn:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256
2. There are 2 locations  within the VLSR appliance that hold a refence to the vCenter Server certificate.  Either of these locations may need to be manually updated.

   1.  The lsppThumbprint parameter in /opt/vmware/dr-client/lib/h5dr.properties:          

    # grep -i lsppThumbprint /opt/vmware/dr-client/lib/h5dr.properties

   2. The thumbprint value in the VLSR database table pd_sslthumbprintstore. Replace 'vc-fqdn' in the below command:

    # echo "SELECT db_id, thumbprint,endpointurl,serviceid  from pd_sslthumbprintstore where endpointurl like '%vc-fqdn%';" | /opt/vmware/vpostgres/current/bin/psql -U srmdb

3. Ensure vCenter FQDN is updated in pd_localsite table as well.
   
   
4. Make a backup of the file before editing. Use copy command: cp /opt/vmware/dr-client/lib/h5dr.properties /opt/vmware/dr-client/lib/h5dr.properties.bak
   
   
5. Edit the /opt/vmware/dr-client/lib/h5dr.properties lsppThumbprint with the new vCenter thumbprint from step 1. Ensure to add the backslash (\) to delimit the colon(:) in thumbprint.
   eg: lsppThumbprint=#\:#\:#\:#\:#\:#\:#\:#\:#\:#\:#
   
   
6. Reboot VLSR appliance, and run Reconfigure.
   NOTE: If reconfigure completes, the pd_sslthumbprintstore table should have updated accordingly.
   
   
7. Verify the thumbprint value in the VLSR database table pd_sslthumbprintstore has updated by running command in VLSR SSH. Replace 'vc-fqdn' in the below command:

   # echo "SELECT db_id, thumbprint,endpointurl,serviceid  from pd_sslthumbprintstore where endpointurl like '%vc-fqdn%';" | /opt/vmware/vpostgres/current/bin/psql -U srmdb

8. If the reconfigure fails or if in step 6 it is verified that the pd_sslthumbprintstore is not updated with the new vCenter thumbprint - Enter SQL mode in the VLSR SSH:

   1. Run this command to enter into srmdb: /opt/vmware/vpostgres/current/bin/psql -U srmdb

   2. Next, list the vCenter entries along with the current thumbprint information in pd_sslthumbprintstore table: srmdb=> select thumbprint from pd_sslthumbprintstore where endpointurl like '%vc-fqdn%';

   3. Update the pd_sslthumbprintstore table entries for the vCenter with the new vCenter thumbprint from step 1: srmdb=> update pd_sslthumbprintstore set thumbprint='#:#:#:#:#:#:#:#:#.....' where endpointurl like '%vc-fqdn%';

   4. Validate the change using query: srmdb=> select thumbprint from pd_sslthumbprintstore where endpointurl like '%vc-fqdn%';   

   5. Exit the database using: Ctrl+D
      
      

9. Reboot VLSR appliance, and run Reconfigure.

The reconfigure task should now be successful and the issue should be resolved.