VMware Live Recovery Error: vSphere Replication Server Inaccessible
search cancel

VMware Live Recovery Error: vSphere Replication Server Inaccessible

book

Article ID: 426484

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms :

  • vCenter UI shows VMware Live Recovery with error on vSphere replication not accessible,

  • Login on Prod VLR UI displays below error,

    Error - "Unable to acquire token from sso Server at "https://FQDN/sdk/dr.vsphere.local"

  • While running reconnect on Site Recovery UI, Error appears as per below screenshot,

Environment

VMware Live Recovery 9.x

Cause

  • The vCenter and Replication Server are unable to communicate because their certificates are out of sync. The vSphere Replication Management (HMS) service cannot start because it is unable to establish a secure connection with vCenter. This is typically caused by a certificate or trust mismatch between the two components.

  • From the logs /opt/vmware/support/logs/hms/hms.log,
    Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and pinned certificates are not configured and thumbprint verification is not configured
    at com.vmware.vim.vmomi.client.http.impl.ClientExceptionTranslator.translate(ClientExceptionTranslator.java:75) ~[vlsi-client-9.0.4.0.jar:?]
    ... 39 more
    Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: SSL handshake from 0.0.0.0/0.0.0.0:54754 to ucs3-s-vc01.ucs.local/###.###.###.###:443 failed in 12 ms
    at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.handleHandshakeException(ThumbprintTrustManager.java:769) ~[vlsi-client-9.0.4.0
    .jar:?]
    Caused by: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
    ####-##-## 16:41:59.775 ERROR com.vmware.hms [main] (..vmware.hms.App) {} [] |
    HMS SERVER ERROR
    org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vrOperationsSender' defined in com.vmware.hms.cfg.spring.PhoneHomeCollectorConfiguration: Unsatisfied dependency expressed through method 'vrOperationsSender' parameter 0: Error creating bean with name 'hmsPhoneHomeCollector' defined in com.vmware.hms.cfg.spring.PhoneHomeCollectorConfiguration: Failed to instantiate [com.vmware.hms.phcollector.HmsPhoneHomeCollector]: Factory method 'hmsPhoneHomeCollector' threw exception with message : com.vmware.vim.vmomi.client.exception.SslException:com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and pinned certificates are not configured and thumbprint verification is not configured

  • The logs indicate trust issue with the certificate chain and thumbprints.

  • lsdoctor script was run and its output is as below,

    root@ucs3-s-vc01 [ /tmp/lsdoctor-251006 ]# python lsdoctor.py -l

        ATTENTION:  You are running a reporting function.  This doesn't make any changes to your environment.
        You can find the report and logs here: /var/log/vmware/lsdoctor

    ####-##-##T17:22:34 INFO main: You are reporting on problems found across the SSO domain in the lookup service.  This doesn't make changes.
    ####-##-##T17:22:35 INFO live_checkCerts: Checking services for trust mismatches...
    ####-##-##T17:22:35 INFO generateReport: Listing lookup service problems found in SSO domain
    ####-##-##T17:22:35 ERROR generateReport: default-first-site\###-#-####.###.##### (vSphere Replication) found SSL Trust Mismatch: Please run python ls_doctor.py--trustfix option on this node.
    ####-##-##T17:22:35 INFO generateReport: No issues detected in the lookup service entries for ###-#-####.###.##### (SRM).
    ####-##-##T17:22:35 INFO generateReport: Report generated:  /var/log/vmware/lsdoctor/###-#-####.###.#####-2026-01-22-172234.json

  • The lsdoctor check confirms trustfix issues exist between the vCenter Server and vSphere Replication.

Resolution

Run below step fails to resolve the thumbprint mismatch issue, use the lsdoctor script as follows:

  1. Download the lsdoctor script from KB-80469

  2. Upload the script to the vCenter Server.

  3. Navigate to the directory where the script was uploaded.

  4. Extract the contents of the ZIP file:

    • unzip lsdoctor.zip

  5. Run the below command for issue check:

    • python lsdoctor.py --l

  6. Run the following command to fix the thumbprint mismatch (if reported):

    • python lsdoctor.py --trustfix

    • Output : 
      root@###-#-#### [ /tmp/lsdoctor-251006 ]# python lsdoctor.py -t
          WARNING:  This script makes permanent changes.  Before running, please take *OFFLINE* snapshots
          of all VC's and PSC's at the SAME TIME.  Failure to do so can result in PSC or VC inconsistencies.
          Logs can be found here: /var/log/vmware/lsdoctor
      ####-##-##T17:24:25 INFO main: You are checking for and fixing SSL trust mismatches in the local SSO site.  NOTE:  Please run this script one PSC or VC per SSO site.
      Have you taken offline (PSCs and VCs powered down at the same time) snapshots of all nodes in the SSO domain or supported backups?[y/n]y

      Provide password for [email protected]:
      ####-##-##T17:24:51 INFO __init__: Retrieved services from SSO site: Default-First-Site
      ####-##-##T17:24:51 INFO findAndFix: Checking services for trust mismatches...
      ####-##-##T17:24:51 INFO findAndFix: Attempting to reregister ########-####-####-####-############ for ###-#-####.###.#####
      ####-##-##T17:25:00 INFO findAndFix: We found 42 mismatch(s) and fixed them :)
      ####-##-##T17:25:00 INFO main: Please restart services on all PSC's and VC's when you're done. 

  7. After executing the script, restart the vCenter services to complete the fix.

    • service-control --stop --all && service-control --start --all

  8. Initiate the reconfiguration process on the VLR appliances at both sites. Following this, verify that the 'HMS' service and all other dependencies have successfully started.

    Note: Please capture VLR and VC Snapshots before performing appliance reconfiguration.

    Refer to the document to reconfigure VLR - Reconfigure the VMware Live Recovery Appliance

    Perform reconnect of Site Pair post reconfiguring SRM's, refer - Reconnect the Connection Between Sites.

  9. From the SRM UI, use the 'Reconnect' option to re-establish the site pairing.
Powered by Wolken Software