vSphere Replication: hbrsrv Service Fails with "Couldn't Initialize Broker Token Manager Crypto!" Backtrace

Products

• VMware Live Recovery

Issue/Introduction

This article addresses the issue where the `hbrsrv` service in vSphere Replication fails to start. This failure prevents connectivity to the vSphere Replication service on port 8123.

Symptoms:

When `hbrsrv` fails to start, you may encounter the following symptoms:

• Inability to connect to vSphere Replication on port 8123
• The `hbrsrv` logs may show entries similar to the following:

  xxx verbose hbrsrv[04839] [Originator@6876 sub=Crypto] Loading Private Key '/etc/vmware/ssl/hbrsrv_broker_private.pem'
  xxx verbose hbrsrv[04839] [Originator@6876 sub=Crypto] Loading Public Key '/etc/vmware/ssl/hbrsrv_broker_public.pem'
  xxx verbose hbrsrv[04839] [Originator@6876 sub=Crypto] Computing key expiration from private key file modification time.
  xxx panic hbrsrv[04839] [Originator@6876 sub=Main] HbrError stack:
  xxx panic hbrsrv[04839] [Originator@6876 sub=Main]    [0] Could not compute a reasonable expiration time for private key '/etc/vmware/ssl/hbrsrv_broker_private.pem'
  xxx panic hbrsrv[04839] [Originator@6876 sub=Main]    [1] Now: 2023-12-04T08:11:52.24508Z Creation time: 2023-05-05T03:34:26Z
  xxx panic hbrsrv[04839] [Originator@6876 sub=Default]
  --> Panic: Couldn't initialize broker token manager crypto!
  --> Backtrace:
  --> [backtrace begin] product: VMware vSphere Replication Server, version: 8.0.1, build: build-21471995, tag: hbrsrv, cpu: x86_64, os: linux, buildType: release
  --> backtrace[00] hbrsrv-bin[0x00BBE95F]
  --> backtrace[01] hbrsrv-bin[0x00BB4CE2]
  --> backtrace[02] hbrsrv-bin[0x00CEA297]
  --> backtrace[03] hbrsrv-bin[0x00CEA3B1]
  --> backtrace[04] hbrsrv-bin[0x00996FD5]
  --> backtrace[05] hbrsrv-bin[0x00903EC3]
  --> backtrace[06] hbrsrv-bin[0x007D3F99]
  --> backtrace[07] libc.so.6[0x00022B87]
  --> backtrace[08] hbrsrv-bin[0x008A2FC1]
  --> [backtrace end]

Environment

VMware vSphere Replication 8.x

VMware vSphere Replication 9.x

Cause

The issue is caused by discrepancies in the timestamp of the certificate files, which prevents proper computation of the key expiration.

This results in the failure of the `hbrsrv` service to initialize the broker token manager cryptography.

Resolution

To resolve this issue, follow these steps:

1. Snapshot VR

• Create a snapshot of the vSphere Replication (VR) environment to ensure that you can revert to the current state if needed.

2. Remove Old PEM Files

• Delete the existing SSL certificate files to allow the `hbrsrv` service to generate new ones:
    
     rm /etc/vmware/ssl/hbrsrv_broker_private.pem
   rm /etc/vmware/ssl/hbrsrv_broker_public.pem

3. Restart the hbrsrv Service

   systemctl stop hbrsrv
     systemctl start hbrsrv

• Restarting the service will create new `hbrsrv_broker_private.pem` and `hbrsrv_broker_public.pem` files.

Additional Information

Impact/Risks:

During this procedure, VM replication will be temporarily unavailable. Ensure to perform these steps during a maintenance window or when replication services can be paused.