vSphere Replication: hbrsrv Service Fails with "Couldn't Initialize Broker Token Manager Crypto!" Backtrace
search cancel

vSphere Replication: hbrsrv Service Fails with "Couldn't Initialize Broker Token Manager Crypto!" Backtrace

book

Article ID: 315200

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

This article addresses the issue where the `hbrsrv` service in vSphere Replication fails to start. This failure prevents connectivity to the vSphere Replication service on port 8123.

Symptoms:


When `hbrsrv` fails to start, you may encounter the following symptoms:

  • Inability to connect to vSphere Replication on port 8123
  • The `hbrsrv` logs(/var/log/vmware/hbrsrv.log) may show entries similar to the following:

  xxx verbose hbrsrv[04839] [Originator@6876 sub=Crypto] Loading Private Key '/etc/vmware/ssl/hbrsrv_broker_private.pem'
  xxx verbose hbrsrv[04839] [Originator@6876 sub=Crypto] Loading Public Key '/etc/vmware/ssl/hbrsrv_broker_public.pem'
  xxx verbose hbrsrv[04839] [Originator@6876 sub=Crypto] Computing key expiration from private key file modification time.
  xxx panic hbrsrv[04839] [Originator@6876 sub=Main] HbrError stack:
  xxx panic hbrsrv[04839] [Originator@6876 sub=Main]    [0] Could not compute a reasonable expiration time for private key '/etc/vmware/ssl/hbrsrv_broker_private.pem'
  xxx panic hbrsrv[04839] [Originator@6876 sub=Main]    [1] Now: 2023-12-04T08:11:52.24508Z Creation time: 2023-05-05T03:34:26Z
  xxx panic hbrsrv[04839] [Originator@6876 sub=Default]
  --> Panic: Couldn't initialize broker token manager crypto!
  --> Backtrace:
  --> [backtrace begin] product: VMware vSphere Replication Server, version: 8.0.1, build: build-21471995, tag: hbrsrv, cpu: x86_64, os: linux, buildType: release
  --> backtrace[00] hbrsrv-bin[0x00BBE95F]
  --> backtrace[01] hbrsrv-bin[0x00BB4CE2]
  --> backtrace[02] hbrsrv-bin[0x00CEA297]
  --> backtrace[03] hbrsrv-bin[0x00CEA3B1]
  --> backtrace[04] hbrsrv-bin[0x00996FD5]
  --> backtrace[05] hbrsrv-bin[0x00903EC3]
  --> backtrace[06] hbrsrv-bin[0x007D3F99]
  --> backtrace[07] libc.so.6[0x00022B87]
  --> backtrace[08] hbrsrv-bin[0x008A2FC1]
  --> [backtrace end]

  •  Due to panic of hbrsrv service /opt/vmware/support partition gets 100%

    Filesystem                       Size  Used Avail Use% Mounted on
    devtmpfs                         4.0M     0  4.0M   0% /dev
    tmpfs                            3.9G   40K  3.9G   1% /dev/shm
    tmpfs                            1.6G  720K  1.6G   1% /run
    tmpfs                            4.0M     0  4.0M   0% /sys/fs/cgroup
    /dev/sda4                         14G  3.6G  9.2G   29% /
    tmpfs                            3.9G  164K  3.9G   1% /tmp
    /dev/sda2                        238M   35M  191M  16% /boot
    /dev/mapper/support_vg-heapdump  3.9G   96K  3.7G   1% /opt/vmware/heapdump
    /dev/mapper/support_vg-logs      975M  192M  733M  21% /opt/vmware/logs
    /dev/mapper/support_vg-vrmsdb    9.8G  181M  9.1G   2% /var/lib/vrmsdb
    /dev/mapper/support_vg-support   2.0G  2.0G     0 100% /opt/vmware/support
    tmpfs                            795M     0  795M   0% /run/user/666

Environment

VMware vSphere Replication 8.x
VMware vSphere Replication 9.x

Cause

The issue is caused by discrepancies in the timestamp of the certificate files, which prevents proper computation of the key expiration.

This results in the failure of the `hbrsrv` service to initialize the broker token manager cryptography.

Resolution

To resolve this issue, perform the following steps:

  1. Take a Snapshot of the vSphere Replication Appliance

    Create a snapshot of the vSphere Replication (VR) appliance to ensure that you can revert to the current state if required.

  2. Check /opt/vmware/support Partition Usage

    • If the /opt/vmware/support partition is 100% utilized, disable the hbrsrv service using the following command, then proceed to Step 3:

       
       systemctl disable hbrsrv.service

    • If the partition is not full, proceed directly to Step 3.

  3. Remove Old PEM Certificate Files

    Delete the existing SSL certificate files so that the hbrsrv service can regenerate new certificates:

     
          rm /etc/vmware/ssl/hbrsrv_broker_private.pem
          rm /etc/vmware/ssl/hbrsrv_broker_public.pem

    If the hbrsrv service was disabled in Step 2, re-enable it using the following command:

          systemctl enable hbrsrv.service

  4. Restart the hbrsrv Service

    Restart the service to apply the changes

            systemctl stop hbrsrv

          systemctl start hbrsrv

    After the service restarts successfully, new hbrsrv_broker_private.pem and hbrsrv_broker_public.pem files are created automatically.

Additional Information

Impact/Risks:

During this procedure, VM replication will be temporarily unavailable. Ensure to perform these steps during a maintenance window or when replication services can be paused.

Powered by Wolken Software