The task “Re-balancing replications between Hbrsrvuws” in vCenter keeps failing
search cancel

The task “Re-balancing replications between Hbrsrvuws” in vCenter keeps failing

book

Article ID: 418041

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

  • The task “Re-balancing replications between Hbrsrvuws” in vCenter fails with the error:
      "There have been errors during vSphere Replication Servers load balancer execution, please refer to the vSphere Replication Management Server log for detailed information."

  • The “Re-balancing replications between Hbrsrvuws” task keeps running repeatedly and fails each time.

  • The task continues to run in vCenter even after disabling the "hbrsrv-monitor-rebalance-enabled" configuration parameter (by setting it to false in the /opt/vmware/etc/hms/hms-configuration.xml file on the HMS server) or after the HMS server is powered off.

Environment

  • VMware Live Recovery Appliance 9.x
  • vCenter Server 8.x

Cause

  • The task fails because the initiator user account used for authorization is stale in vCenter.

  • From the /var/log/vmware/hms/hms.0000*.log, confirm the service account  which the vSphere Replication Management Server (HMS) is trying to authenticate with the vCenter:

2025-10-27 14:34:26.398 INFO  vmware.jvsl.sessions [hms-main-scheduled-thread-10] (..impl.vc.SsoLoginHandler$1) {operationID=mBZ-oGqfGPf-HMS} [] | renewToken for {Name: com.vmware.vr-sa-10dbdf16-####-####-####-0018054fe158, Domain: VSPHERE.LOCAL} (id:_aacbd0f7-d7d3-43c6-85be-4752c22ddcea; exp:Tue Oct 28 14:34:24 IST 2025)
2025-10-27 14:34:26.399 DEBUG security.authentication.sessionmap [tcweb-3] (..security.authentication.SessionMap) {operationID=mBZ-oGqfGPf-HMS} [] | Adding new session to the session map:com.vmware.hms.security.authentication.SsoUserSession@36985932:[
        com.vmware.vim.binding.hms.UserSession:
        key = site_10d...0bb25fa6:
        sessionID=b2bb5ca,
        userName = com.vmware.vr-sa-10dbdf16-375e-4c66-9b77-0018054fe158@VSPHERE.LOCAL,
        fullName = com.vmware.vr-sa-10dbdf16-375e-4c66-9b77-0018054fe158 (peer '10dbdf16-375e-4c66-9b77-0018054fe158'),
        loginTime = Mon Oct 27 14:34:26 IST 2025,
        lastActiveTime = Mon Oct 27 14:34:26 IST 2025

This above log line confirms the Service Account which the HMS is using for token renewal.

  • In the /var/log/vmware/vpxd-svc/vpxd-svc.log, the initiator user for the “Re-balancing replications between Hbrsrvuws” task shows privilege-related errors:

2025-11-07T08:33:30.230Z [dataservice-7 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=9611e526-e500-4925-89b8-0535bd3cbe73 IS] User VSPHERE.LOCAL\com.vmware.vr-sa-ab1ca80c-####-####-####-aa17b9242c15 does not have privileges [System.Read] on object urn%3Avmomi%3AInventoryServiceTag%3A68b21362-adc7-4415-a8b4-6f933adbd46d%3AGLOBAL

This confirms that the task is being executed using an account that no longer has the required privileges.

  • To verify, the service account list check using the dir-cli command:

root@vlr [ /tmp ]# /usr/lib/vmware-vmafd/bin/dir-cli svcaccount list --password '*******' | grep -i vr
18. com.vmware.vr-sa-ab1ca80c-####-####-####-aa17b9242c15
40. com.vmware.vr-sa-10dbdf16-####-####-####-0018054fe158

Among these, multiple stale service accounts will be found that are no longer in use.

Resolution

 

  1. Take an offline snapshot of the vCenter.

  2. Delete all stale vSphere Replication service accounts from vCenter using the dir-cli svcaccount delete command.

  3. Delete any associated entries for these accounts from vCenter UI --> Permissions tab.

Create a Broadcom Support ticket for further validation and assistance with implementing the Resolution.

 

Powered by Wolken Software