vSphere Replication: hbrsrv Service Fails with "Couldn't Initialize Broker Token Manager Crypto!" Backtrace
search cancel

vSphere Replication: hbrsrv Service Fails with "Couldn't Initialize Broker Token Manager Crypto!" Backtrace

book

Article ID: 315200

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

This article addresses the issue where the `hbrsrv` service in vSphere Replication fails to start. This failure prevents connectivity to the vSphere Replication service on port 8123.

Symptoms:

  • vSphere Replication shows as disconnected in Site Recovery UI

  • Reconnecting it throws the error:

ERROR
Operation Failed
Cannot establish a TCP connection to server at '10.X.X.X:8123'. Details: 'https://10.X.X.X:8123 invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.##.##.##:8123 [/10.X.X.X] failed: Connection refused (Connection refused)"'.

  • Review the ESXi host logs and confirm repeated connection refusal errors toward the VR server:

2026-05-13T05:43:33.487Z cpu22:12390744)WARNING: Hbr: 574: Connection failed to 10.##.#.### (groupID=GID-60d89128-9d81-4346-b721-#######): Connection refused
2026-05-13T05:43:33.487Z cpu22:12390744)WARNING: Hbr: 5093: Failed to establish connection to [10.##.#.###]:31031 (groupID=GID-60d89128-9d81-4346-b721-#########): Connection refused

  • Review HMS logs under /opt/vmware/hms/logs/ and identify local service connectivity failures:

2024-01-26 15:09:09.042 ERROR hms.net.hbr.ping.svr.52d8c50d-9d4c-2c51-2d0f-#######[hms-ping-scheduled-thread-8] (..net.impl.VmomiPingConnectionHandler) [operationID=d82ac874-74b9-43bc-9b68-32bcee38c3dd-HMS-PING---Ping Thread for session key: N/A and vmomi session: null and server: 10.##.##.##:8123] | Ping for server 10.##.##.##:8123 for session: N/A failed: com.vmware.vim.vmomi.client.exception.ConnectionException: https://10.##.##.##:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.##.##.##:8123 [/10.##.##.##] failed: Connection refused (Connection refused)" : https://10.##.##.##:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.##.##.##:8123 [/10.##.##.##] failed: Connection refused (Connection refused)"

  • Verify the hbrsrv service state using systemd and confirm that the service repeatedly crashes during initialization:

root [ /]# systemctl status hbrsrv
● hbrsrv.service - Host-based replication server.
     Loaded: loaded (/usr/lib/systemd/system/hbrsrv.service; disabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: core-dump) since Wed 2026-05-13 10:12:19 UTC; 7s ago
    Process: 453886 ExecStartPre=/usr/bin/chown hbrsrv:vmware /etc/vmware/ssl/hbrsrv.key /etc/vmware/ssl/hbrsrv.crt (code=exited, status=0/SUCCESS)
    Process: 453888 ExecStartPre=/usr/bin/hbrsrv-set-ip-for-filter.sh (code=exited, status=0/SUCCESS)
    Process: 453893 ExecStartPre=/usr/bin/hbrsrv-set-nic-config.sh (code=exited, status=0/SUCCESS)
    Process: 453909 ExecStart=/usr/bin/hbrsrv --daemon --pidfile /var/run/hbrsrv/hbrsrv.pid --vmodlport $HBRSRV_VMODL_PORT --lwdport $HBRSRV_LWD_PORTS --lwdsport $HBRSRV_LWDS_PORTS (code=e>
   Main PID: 453914 (code=dumped, signal=ABRT)

May 13 10:12:19 localhost systemd[1]: hbrsrv.service: Failed with result 'core-dump'.

  • Due to panic of hbrsrv service /opt/vmware/support partition gets 100%

Filesystem                       Size  Used Avail Use% Mounted on
devtmpfs                         4.0M     0  4.0M   0% /dev
tmpfs                            3.9G   40K  3.9G   1% /dev/shm
tmpfs                            1.6G  720K  1.6G   1% /run
tmpfs                            4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sda4                         14G  3.6G  9.2G   29% /
tmpfs                            3.9G  164K  3.9G   1% /tmp
/dev/sda2                        238M   35M  191M  16% /boot
/dev/mapper/support_vg-heapdump  3.9G   96K  3.7G   1% /opt/vmware/heapdump
/dev/mapper/support_vg-logs      975M  192M  733M  21% /opt/vmware/logs
/dev/mapper/support_vg-vrmsdb    9.8G  181M  9.1G   2% /var/lib/vrmsdb
/dev/mapper/support_vg-support   2.0G  2.0G     0 100% /opt/vmware/support
tmpfs                            795M     0  795M   0% /run/user/666

Environment

  • VMware vSphere Replication 8.x
  • VMware vSphere Replication 9.x

Cause

  • The issue occurs because the vSphere Replication Server (hbrsrv) cannot initialize the broker token manager crypto component due to an invalid or unreasonable expiration calculation for the broker private key file: /etc/vmware/ssl/hbrsrv_broker_private.pem
  • The service computes the broker key expiration based on the private key file modification timestamp. During startup, the calculated lifetime validation fails, causing the hbrsrv process to trigger a panic condition and terminate abnormally.
  • As a result, the hbrsrv service continuously crashes during startup, making the vSphere Replication Server unavailable.
  • The failure is directly confirmed in the hbrsrv logs:

2026-05-13T10:13:35.663Z verbose hbrsrv[454335] [Originator@6876 sub=Crypto] Loading Public Key '/etc/vmware/ssl/hbrsrv_broker_public.pem'
2026-05-13T10:13:35.665Z verbose hbrsrv[454335] [Originator@6876 sub=Crypto] Computing key expiration from private key file modification time.
2026-05-13T10:13:35.665Z panic hbrsrv[454335] [Originator@6876 sub=Main] HbrError stack:
2026-05-13T10:13:35.665Z panic hbrsrv[454335] [Originator@6876 sub=Main]    [0] Could not compute a reasonable expiration time for private key '/etc/vmware/ssl/hbrsrv_broker_private.pem'
2026-05-13T10:13:35.665Z panic hbrsrv[454335] [Originator@6876 sub=Main]    [1] Now: 2026-05-13T10:13:35.665573Z Creation time: 2025-10-15T11:33:08Z
2026-05-13T10:13:35.667Z panic hbrsrv[454335] [Originator@6876 sub=Default]
-->
--> Panic: Couldn't initialize broker token manager crypto!
--> Backtrace:
--> [backtrace begin] product: VMware vSphere Replication Server, version: 8.0.3, build: build-23167363, tag: hbrsrv, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] hbrsrv-bin[0x00AFF3A2]
--> backtrace[01] hbrsrv-bin[0x00AF150C]
--> backtrace[02] hbrsrv-bin[0x00C8E3E8]
--> backtrace[03] hbrsrv-bin[0x00C8E4F2]
--> backtrace[04] hbrsrv-bin[0x008B33AA]
--> backtrace[05] hbrsrv-bin[0x0081E89B]
--> backtrace[06] hbrsrv-bin[0x006DC7DA]
--> backtrace[07] libc.so.6[0x00027F0A]
--> backtrace[08] hbrsrv-bin[0x007B6D75]
--> backtrace[09] (no module)
--> [backtrace end]
(END)

Resolution

To resolve this issue, perform the following steps:

  1. Take a Snapshot of the vSphere Replication Appliance

    Create a snapshot of the vSphere Replication (VR) appliance to ensure that you can revert to the current state if required.

  2. Check /opt/vmware/support Partition Usage

    • If the /opt/vmware/support partition is 100% utilized, disable the hbrsrv service using the following command, then proceed to Step 3:

       
       systemctl disable hbrsrv.service

    • If the partition is not full, proceed directly to Step 3.

  3. Remove Old PEM Certificate Files

    Delete the existing SSL certificate files so that the hbrsrv service can regenerate new certificates:

     
          rm /etc/vmware/ssl/hbrsrv_broker_private.pem
          rm /etc/vmware/ssl/hbrsrv_broker_public.pem

    If the hbrsrv service was disabled in Step 2, re-enable it using the following command:

          systemctl enable hbrsrv.service

  4. Restart the hbrsrv Service

    Restart the service to apply the changes

            systemctl stop hbrsrv

          systemctl start hbrsrv

    After the service restarts successfully, new hbrsrv_broker_private.pem and hbrsrv_broker_public.pem files are created automatically.

Additional Information

Impact/Risks:

During this procedure, VM replication will be temporarily unavailable. Ensure to perform these steps during a maintenance window or when replication services can be paused.
vSphere Replication: hbrsrv サービスが "Couldn't Initialize Broker Token Manager Crypto!" のバックトレースを出力し起動失敗

Powered by Wolken Software