"Failed to establish connection to HBR" error after upgrading the Replicator appliance in Cloud Director Availability 4.x

Products

• VMware Cloud Director

Issue/Introduction

• After upgrading a Cloud Director Availability Replicator appliance, you see an error similar to:

Failed to establish connection to HBR

• When you run the command journalctl -u hbrsrc.service -b on the Replicator appliance, you see messages similar to:

Jul 18 13:09:17 example.local hbrsrv.stdout[717349]: ERROR: C++ run-time terminate handler invoked.. Check syslog or logfile for details Jul 18 13:09:17 Replicator_FQDN hbrsrv[717348]: C++ ###-#### terminate handler invoked. Jul 18 13:09:17 example.local hbrsrv[717348]: Unhandled exception 'Vmacore::FileFormatException': Duplicate child: hostConnectionUseSdkEndpoint Jul 18 13:09:17 example.local hbrsrv.stdout[717349]: Unhandled exception 'Vmacore::FileFormatException': Duplicate child: hostConnectionUseSdkEndpoint Jul 18 13:09:17 example.local systemd[1]: hbrsrv.service: Main process exited, code=dumped, status=6/ABRT Jul 18 13:09:17 example.local systemd[1]: hbrsrv.service: Failed with result 'core-dump'.

• On the file /etc/vmware/hbrsrv.xml in the replicator appliance you see duplicated entries at the end of the file similar to:
  
  <extendDiskPITHierarchyPolicy>preserve</extendDiskPITHierarchyPolicy>
  <removeMPITsBeforeBaseDisks>false</removeMPITsBeforeBaseDisks>
  <backupMode>backup_mode_on</backupMode>
  <backupCriticalMode>true</backupCriticalMode>
  <useDatastoreTickets>false</useDatastoreTickets>
  <hostConnectionUseSdkEndpoint>false</hostConnectionUseSdkEndpoint>
  <hostConnectionUseHbrTunnelEndpoint>false</hostConnectionUseHbrTunnelEndpoint>
  <hostConnectionUseHbrTunnelEndpoint>false</hostConnectionUseHbrTunnelEndpoint>
  <hostConnectionUseSdkEndpoint>false</hostConnectionUseSdkEndpoint>

Environment

Cloud director Availability 4.x

Cause

This issue occurs due to a parameter duplication in the HBR configuration file generated during the replicator upgrade.

Resolution

To fix this issue follow the below steps:

1. SSH to the replicator appliance
2. Stop the HBR service 
   
   systemctl stop hbrsrv.service
   
   
3. backup the configuration file 
   
   cp /etc/vmware/hbrsrv.xml /etc/vmware/hbrsrv.xml.bkp
   
   
4. Modify /ect/vmware/hbrsrv.xml so there is only one entry for each property. Following the above example you should delete or comment one of the lines for the repeated ones so it would look similar to this:
   
   <extendDiskPITHierarchyPolicy>preserve</extendDiskPITHierarchyPolicy>
<removeMPITsBeforeBaseDisks>false</removeMPITsBeforeBaseDisks>
<backupMode>backup_mode_on</backupMode>
<backupCriticalMode>true</backupCriticalMode>
<useDatastoreTickets>false</useDatastoreTickets>
<hostConnectionUseSdkEndpoint>false</hostConnectionUseSdkEndpoint>
<hostConnectionUseHbrTunnelEndpoint>false</hostConnectionUseHbrTunnelEndpoint>
<!-- <hostConnectionUseHbrTunnelEndpoint>false</hostConnectionUseHbrTunnelEndpoint> -->
<!-- <hostConnectionUseSdkEndpoint>false</hostConnectionUseSdkEndpoint> -->
   
   
5. Restart the HBR service
   
   systemctl start hbrsrv.service
   
   
6. Check if the error in the Replicator appliance interface is fixed.