HMS Service Failure and Site Pairing Issues After VRMS 8.x Upgrade Due to Disabled Embedded HBR Server
search cancel

HMS Service Failure and Site Pairing Issues After VRMS 8.x Upgrade Due to Disabled Embedded HBR Server

book

Article ID: 386839

calendar_today

Updated On:

Products

VMware Live Recovery VMware vSphere ESXi

Issue/Introduction

Symptoms:

  1. After upgrading to VMware VRMS version 8.8.0.2 and reconfiguring, users are unable to establish site pair.
  2. The HMS service repeatedly fails and restarts
  3. CPU utilization of the appliance spikes to 100%.

When opening the Site Recovery UI, the site pair fails with the error message:

"Unable to retrieve pairs from extension server at https://vr-appliance:8043 . Unable to connect to the HBR Management Server at https://vr-appliance:8043. Reason Java.net.ConnectionException: Connection Refused  & VR management server at vr-appliance is unreachable"

From opt/vmware/hms/logs/hms.log we observe below error messages

YYYY-MM-DD HH:MM:SS.SSS ERROR com.vmware.hms.HmsService [hms-main-thread-1] (..vmware.hms.HmsService) [] | stage 2 starting...FAILED
HMS Server failed to start successfully:
java.lang.IllegalStateException: Cannot retrieve HBR broker since embedded HBR server is disabled.

From /opt/vmware/hms/logs/hms-configtool.log we observe below error messages

YYYY-MM-DD HH:MM:SS.SSS WARN com.vmware.hms.configtool.App [main] (..hms.configtool.DatabaseProcessor) [] | Error while executing SQL statement: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
YYYY-MM-DD HH:MM:SS.SSS ERROR com.vmware.hms.configtool.App [main] (..hms.configtool.DatabaseProcessor$JdbcRunnable) [] | Cannot execute sql statement.
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
YYYY-MM-DD HH:MM:SS.SSS TRACE hms.hbr.monitor [hms-ping-scheduled-thread-0] (..impl.hbr.HbrConnectionMonitor) [operationID=57e2fac7-2168-48e8-a511-1c409d781631-HMS-PING] | Connection status for VR Server 'VR-appliance' is updated to 'false' in the database.

Environment

vSphere Replication 8.x
vSphere Replication 9.x - See important note in Cause section

 

Cause

The issue occurs because, after the upgrade, the embedded HBR server is disabled. The HMS service cannot retrieve the HBR broker, leading to the failures observed in the logs and the inability to establish site pairing. Improper configuration issues may prevent the services from initializing correctly.

The /opt/vmware/hms/conf/hms-configuration.xml file has the "hms-embedded-hbr" value set to "false".

NOTE: In vSphere replication 9.0 and higher this option is not compatible with enhanced replication, if scale-out-mode is set to true (enhanced replication is enabled) and hms-embedded-hbr is set to false a similar error "Unable to connect to the HBR Management Server" will be observed.

Resolution

 To resolve this issue, follow these steps:

  1. Take a snapshot of the vSphere Replication Appliance.
  2. Verify if the value of hms-embedded-hbr using below command:
    # grep -i hms-embedded-hbr /opt/vmware/hms/conf/hms-configuration.xml
  3. The embedded HBR server must be enabled in the configuration to allow the HMS service to function correctly.
  4. Using the vi editor edit the hms-configuration.xml file and change the value of the hms-embedded-hbr parameter to true.
  5. After updating the configuration, restart the HMS service for the changes to take effect
    # systemctl restart hms      
Powered by Wolken Software