Enhanced Replication Mapping fails and VM replications reported as "Not Active (RPO Violation)"
search cancel

Enhanced Replication Mapping fails and VM replications reported as "Not Active (RPO Violation)"

book

Article ID: 394533

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

  • Enhanced Replication Mappings throws following error:

    "Fault occurred when performing health check. Details: 'Connect: certificate verify failed (SSL routines)":

  • In some cases, the enhanced replication mappings will be in error state, but all the connections return a status of good on the site recovery UI. However, /opt/vmware/hms/logs/hms.log file, reports below errors.

    2026-04-08 09:12:27.766 INFO  com.vmware.hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [tcweb-7] (..response.filter.I18nActivationResponseFilter) [operationID=d222215b-d436-403b-afc7-d29d852193ac-HMS-5667064,sessionID=03C85423] | The localized message is: Fault occurred while performing health check. Details: 'Connect: certificate verify failed (SSL routines)'.
    2026-04-08 09:12:27.766 INFO  com.vmware.hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [tcweb-7] (..response.filter.I18nActivationResponseFilter) [operationID=d222215b-d436-403b-afc7-d29d852193ac-HMS-5667064,sessionID=03C85423] | The localized message is: Fault occurred while performing health check. Details: 'Connect: certificate verify failed (SSL routines)'.

  • VMs configured for replication using enhanced replication enters a Not Active state

  • ESXi hosts and the vSphere replication appliance uses custom signed certificate and certificates are renewed recently

  • Source ESXi host /var/run/log/hbr-agent.log indicates that the SSL handshake is failing to the target VR appliance

    2026-04-08T09:12:27.766Z In(166) hbr-agent-bin[40670037]: [0x000000d8875c8700] info: [ProxyConnection] Setting up secure tunnel to broker on #.#.#.64:32032
    2026-04-08T09:12:27.766Z In(166) hbr-agent-bin[40670037]: [0x000000d8875c8700] info: [Proxy [Group: ] -> [#.#.#.64:32032]] Bound to vmk: vmk3 for connection to #.#.#.64:32032
    2026-04-08T09:12:27.823Z In(166) hbr-agent-bin[40670037]: [0x000000d887547700] info: [Proxy [Group: ] -> [#.#.#.64:32032]] TCP Connect latency was 56319µs
    2026-04-08T09:12:27.882Z In(166) hbr-agent-bin[40670037]: [0x000000d8874c6700] error: [Proxy [Group: ] -> [#.#.#.64:32032]] SSL handshake failed: certificate verify failed (SSL routines)
    2026-04-08T09:12:27.882Z In(166) hbr-agent-bin[40670037]: [0x000000d8874c6700] error: [Proxy [Group: ] -> [#.#.#.64:32032]] Failed to connect to broker on #.#.#.64:32032: certificate verify failed (SSL routines)
    2026-04-08T09:12:27.882Z In(166) hbr-agent-bin[40670037]: [0x000000d8874c6700] error: [Proxy [Group: ] -> [#.#.#.64:32032]] Failed to connect to broker: certificate verify failed (SSL routines)

  • Target VR server reports following in /var/log/vmware/hbrsrv.log:

    2025-06-11T06:54:03.441Z error hbrsrv[20244] [Originator@6876 sub=Asio] Cannot perform SSL handshake for <TCP '#.#.#.# : 32032'> -> <TCP '#.#.#.# : 52106'> (encrypted): short read
    2025-06-11T06:54:03.441Z error hbrsrv[20244] [Originator@6876 sub=Main] HbrError stack:
    2025-06-11T06:54:03.441Z error hbrsrv[20244] [Originator@6876 sub=Main]    [0] Exception Vmacore::Exception: Cannot perform SSL handshake for <TCP '#.#.#.# : 32032'> -> <TCP '#.#.#.# : 52106'> (encrypted): short read
    2025-06-11T06:54:03.441Z error hbrsrv[20244] [Originator@6876 sub=Main]    [1] Failed HbrSrv accept on socket ([N9HbrServer20BoostTCPServerSocketE:0x000055ef026f9048])

  • Below errors will be reported in some cases:

    A generic error occurred in the vSphere Replication Management Server. Exception details: 'Unable to remove existing protection groups and images from VR Server 'xxx''. Cause: A generic error occurred in the vSphere Replication Management Server. Exception details: 'com.vmware.vim.binding.hbr.replica.fault.HbrRuntimeFault'. 

 

Environment

vSphere Replication 9.x

Cause

The SSL connection to the target connection broker fails because the vSphere Replication Management Server (HMS) cannot verify the broker’s endpoint. This occurs because the Certificate Authority (CA) of the broker is not recognized as trusted by HMS at the time of the connection attempt.

HMS is designed to load the embedded HBR (Host-Based Replication) certificate from the local file path /etc/vmware/ssl/hbrsrv.crt. Once read, this certificate is persisted within the HMS database to facilitate secure communication. 

This issue is triggered by a specific race condition during the system startup sequence. HMS attempts to read the hbrsrv.crt file before the certificate update process has successfully written the current, valid certificate to that file. Consequently, HMS loads and persists an outdated or incomplete certificate, leading to a trust mismatch when it attempts to establish a secure handshake with the connection broker.

Cause Validation:

The target vSphere Replication appliance hbrsrv.log reports unknown certificate authority errors, indicating a failure to validate the connection broker's identity.

2026-04-08T09:06:27.901Z error hbrsrv[3131781] [Originator@6876 sub=Asio] Cannot perform SSL handshake for <TCP '#.#.#.64 : 32032'> -> <TCP '#.#.#.12 : 49152'> (encrypted): tlsv1 alert unknown ca (SSL routines)
2026-04-08T09:06:27.901Z error hbrsrv[3131783] [Originator@6876 sub=Main] HbrError stack:
2026-04-08T09:06:27.901Z error hbrsrv[3131783] [Originator@6876 sub=Main]    [0] Exception Vmacore::Exception: Cannot perform SSL handshake for <TCP '#.#.#.64 : 32032'> -> <TCP '#.#.#.12 : 49152'> (encrypted): tlsv1 alert unknown ca (SSL routines)
2026-04-08T09:06:27.901Z error hbrsrv[3131783] [Originator@6876 sub=Main]    [1] Failed HbrSrv accept on socket ([N9HbrServer20BoostTCPServerSocketE:0x000056493cab7a18])
2026-04-08T09:06:27.901Z info hbrsrv[3131783] [Originator@6876 sub=Main] HbrError stack:
2026-04-08T09:06:27.901Z info hbrsrv[3131783] [Originator@6876 sub=Main]    [0] Exception Vmacore::Exception: Cannot perform SSL handshake for <TCP '#.#.#.64 : 32032'> -> <TCP '#.#.#.12 : 49152'> (encrypted): tlsv1 alert unknown ca (SSL routines)
2026-04-08T09:06:27.901Z info hbrsrv[3131783] [Originator@6876 sub=Main]    [1] Failed HbrSrv accept on socket ([N9HbrServer20BoostTCPServerSocketE:0x000056493cab7a18])
2026-04-08T09:06:27.901Z info hbrsrv[3131783] [Originator@6876 sub=Main]    [2] Ignored error.

Resolution

This issue caused by the race condition will be addressed in VR 9.1

As a workaround, restart the HMS and the HBRSRV services on both vSphere Replication appliances and reconnect them. Once done the HMS service will pick up the new HBR embedded certificate.

You can restart these services from the VAMI 

  • Log into the vSphere Replication VAMI (https://VRMS-IP:5480)
  • Navigate to services
  • Select hms / hbrsrv
  • Select restart

Alternatively, you can open an SSH session to the vSphere Replication appliance

  • systemctl restart hms
  • systemctl restart hbrsrv

Once this is done, 'Reconfigure' the VM replications that showed 'Not Active' status. The replication would sync and then changes the state to 'OK'.

Additional Information

In some instances, the VR appliances need to be powered off/on (not just restarted) to allow vCenter to recreate the appliance environment and push the new certificates.

Powered by Wolken Software