Cannot configure virtual machines for vSphere Enhanced Replication
search cancel

Cannot configure virtual machines for vSphere Enhanced Replication

book

Article ID: 379468

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

Reported errors thrown when a VM participating in an Enhanced Replication configuration is unable to replicate.

No target datastores are accessible from hosts that support enhanced replication.

The virtual machine host does not support enhanced replication.

A replication error occurred at the vSphere Replication Server for replication X . Details: 'No connection to VR Server for virtual machine Y on host Z  in cluster  A: Unknown'.

Replication job "Not Active".

    Synchronization monitoring has stopped. Please verify replication traffic connectivity between the source host and the target vSphere Replication Server. Synchronization monitoring will resume when connectivity issues are resolved.

         

Issue Validation:

  • /opt/vmware/hms/logs/hms.log,
  • The below logs show connection issues with 32032 port.
2025-02-13T23:35:02.499Z Wa(180) vmkwarning: cpu6:39307697)WARNING: Hbr: 2389: Failed to receive handshake response2025-02-13T23:35:02.499Z Wa(180) vmkwarning: cpu6:39307697)WARNING: Hbr: 5362: Failed to establish connection to [xxx.x.x.1]:32032 (groupID=GID-xxxxxxxx-xxxx-xxxx-xxxx-xxxxe421bd77): Broken pipe2025-02-13T23:35:24.472Z In(182) vmkernel: cpu14:2098032)Unmap6: 10536: xxxxxxxxxxxxxxx-datastore1: Acquired UC with 0 OP ucOffset 368640 ucIndex: 752025-02-13T23:35:24.472Z In(182) vmkernel: cpu14:2098032)Unmap6: 10568: xxxxxxxxxxxxxxx-datastore1: Unmap capability 0 OP i: 0 offset: 65536 endROffset: 589824 Acquired: TRUE
  • /var/log/hbr-agent.log,
  • The connection between source Esxi host to Destination VR server on port 32032 fails as seen below.
2025-02-13T23:41:02.513Z In(166) hbr-agent-bin[14752884]: [0x000000342d31f700] error: [Proxy [Group: ] -> [xxx.x.x.7:32032]] Failed to connect to broker: Input/output error
2025-02-13T23:41:17.508Z In(166) hbr-agent-bin[14752884]: [0x000000342d31f700] info: [ConfigManager] No user configuration for key=hbrsvc_target_info in ConfigStore.
2025-02-13T23:41:17.508Z In(166) hbr-agent-bin[14752884]: [0x000000342d31f700] error: [ConfigManager] Failed to get config store object. Comp: esx, Grp: services, Key: hbrsvc_target_info, Id: xx.xx.xx7.208, Prop: certificate
2025-02-13T23:41:17.508Z In(166) hbr-agent-bin[14752884]: [0x000000342d31f700] info: [ProxyConnection] Setting up secure tunnel to broker on xx.xx.xx7.208:32032
2025-02-13T23:41:17.508Z In(166) hbr-agent-bin[14752884]: [0x000000342d31f700] info: [Proxy [Group: ] -> [xx.xx.xx7.208:32032]] Bound to vmk: vmk2 for connection to xx.xx.xx7.208:32032
2025-02-13T23:42:32.514Z In(166) hbr-agent-bin[14752884]: [0x000000342d29e700] error: [Proxy [Group: ] -> [xx.xx.xx7.208:32032]] Failed to connect to xx.xx.xx7.208:32032. Using nic 'vmk2'. Error: Connection timed out
2025-02-13T23:42:32.514Z In(166) hbr-agent-bin[14752884]: [0x000000342d29e700] error: [Proxy [Group: ] -> [xx.xx.xx7.208:32032]] Failed to bind to any of the specified VMKs for connection to xx.xx.xx7.208:32032
2025-02-13T23:42:32.514Z In(166) hbr-agent-bin[14752884]: [0x000000342d29e700] error: [Proxy [Group: ] -> [xx.xx.xx7.208:32032]] Failed to connect to broker on 10.xx.xx.xx7.208:32032: Input/output error
2025-02-13T23:42:32.514Z In(166) hbr-agent-bin[14752884]: [0x000000342d29e700] error: [Proxy [Group: ] -> [xx.xx.xx7.208:32032]] Failed to connect to broker: Input/output error
2025-02-13T23:42:47.511Z In(166) hbr-agent-bin[14752884]: [0x000000342d31f700] info: [ConfigManager] No user configuration for key=hbrsvc_target_info in ConfigStore.
2025-02-13T23:42:47.511Z In(166) hbr-agent-bin[14752884]: [0x000000342d31f700] error: [ConfigManager] Failed to get config store object. Comp: esx, Grp: services, Key: hbrsvc_target_info, Id: xx.xx.xx7.208, Prop: certificate
2025-02-13T23:42:47.511Z In(166) hbr-agent-bin[14752884]: [0x000000342d31f700] info: [ProxyConnection] Setting up secure tunnel to broker on xx.xx.xx7.208:32032
2025-02-13T23:42:47.511Z In(166) hbr-agent-bin[14752884]: [0x000000342d31f700] info: [Proxy [Group: ] -> [xx.xx.xx7.208:32032]] Bound to vmk: vmk2 for connection to xx.xx.xx7.208:32032

Environment

vSphere Replication 9.x 

Cause

Failure to adhere to the network configuration and port requirements will cause issues with replications. 

Resolution

Prerequisites :

  • Enhanced replications require minimum versions vSphere Replication 9.0.x, vCenter Server 8.0u2, and ESXi host 8.0u2 on the target site when the target is an on-premises SDDC.

  • vSphere Replication 9.0 requires network encryption for Enhanced replications.

  • Enhanced replications require TCP network connectivity on ports 31031 and 32032 from the ESXi hosts on which the replicated VMs are running to the ESXi hosts of the cluster containing the target datastore. Make sure your firewall settings are adjusted accordingly.

  • Ensure timestamp is in sync with all env nodes, VR, ESXi, VC

 

 

 

          How do I reconfigure existing replications to use vSphere Replication with enhanced replication capabilities

 

          vSphere Replication Enhanced Replication Mappings

 

Steps to Isolate and Resolve:

To use the enhanced replication capabilities with vSphere Replication, you must reconfigure your existing replication to use Enhanced replication mode.

      • Enhanced replications require minimum versions vSphere Replication 9.0.x, vCenter Server 8.0u2, and ESXi host 8.0u2 on the target site when the target is an on-premises SDDC.
        Product Interoperability Matrix

      • vSphere Enhanced Replication 9.0 requires network encryption for Enhanced replications.


            • Ensure the vmware-hbr-agent vib is installed and running on all ESXi hosts.

            • Check for vib from UI
            • On ESXi host>Services>Configure check for hbr-agent and hbrsrv status

              Check for vib from cli
              # chkconfig --list | grep -i hbr     

              # localcli software vib list | grep -i hbr
                   vmware-hbrsrv                  8.0.2-0.0.22380479                    VMware  VMwareCertified   2024-01-04    host
                   vmware-hbr-agent             8.0.1-0.0.21471995                    VMware  VMwareCertified   2023-07-18    host

               # localcli software vib get | grep -i hbr
               # localcli software vib list  | grep -i hbr


              Installing vib from vSphere Replication 8.5.0.3 onwards, the HBR Agent VIB will no longer be installed automatically on hosts managed by vLCM clusters. However, vSphere Replication will continue to install the HBR Agent VIB on all standalone hosts and hosts, which are a part of non-vLCM managed clusters. This change has been implemented to prevent vSphere Replication from interfering with vLCM’s operations. For hosts not part of vLCM clusters it will continue to use VUM to install the HBR Agent VIB.

               # localcli software vib install -v /tmp/vmware-hbr-agent-8.0.3-0.0.23167363.i386.vib  


              # chkconfig hbr on|off 

 

      • Enhanced replications require TCP network connectivity on ports 31031 and 32032 from the ESXi hosts on which the replicated VMs are running to the ESXi hosts of the cluster containing the target datastore. Make sure your firewall settings are adjusted accordingly. On legacy VR tracffic goes from source ESXi to DR VR, on enhanced replication traffic goes from source ESXi to DR ESXi along with source ESXi to DR VR to bypass.

            • On source ESXi host   
              # nc -z [target-ESX]  32032
              # openssl s_client -connect <IP>:<port>  
                   
                  eg:
                  # openssl s_client -connect esx001b.vcf.lab:32032

               Check routeability between hosts . Run on source ESXi to target ESXi
              # traceroute [target-ESX]

            • On source ESXi host   
              # nc -z [target-VR_server_IP]  32032
              # openssl s_client -connect <IP>:<port>  
                   
                  eg:
                  # openssl s_client -connect Vr_server_IP:32032

 

      • Ensure timestamps is correct on ESXi hosts. Often the NTP daemon had stopped, Run a watch command on ESXi hosts and VR/VC nodes to compare timestamps

          # watch -d date -u 

Additional Information

Powered by Wolken Software