RPO violations occur and replication gets stuck at "Disk consolidation in progress" due to hbrsrv deadlock

Products

• VMware vSphere ESXi

Issue/Introduction

When using Enhanced Replication with a large number of virtual machines, you may observe the following:

• Multiple virtual machines experience intermittent or persistent RPO violations.
• Replication tasks for the affected virtual machines appear stuck at the "Disk consolidation in progress".
• hbrsrv service on the recovery site crashes intermittently due to Out-of-Memory (OOM) conditions.

Environment

vSphere Replication 9.0.x

Cause

This issue is caused by a deadlock condition within the hbrsrv service on the recovery site, triggered by a high concentration of simultaneous replication processes.

The deadlock occurs when a high volume of concurrent replication management tasks (such as RefreshInstances()) saturates the internal consolidation queue.
Since dependent tasks (like disk consolidation or child LROs) are added to the same queue, they cannot be processed because the management tasks are occupying all available threads while waiting for these dependent tasks to finish.

This mutual waiting state results in a deadlock. As accumulated tasks continue to pile up in the queue, it eventually leads to an Out-of-Memory (OOM) condition and causes the service to crash.

Resolution

Broadcom Engineering is aware of this issue and is currently working on a fix for a future release.

As a temporary workaround to mitigate the deadlock risk, perform the following steps to increase the queue and concurrency limits within the hbrsrv configuration via configstorecli.

Workaround

Perform these steps on every ESXi host located at the recovery site.

1. Log in to the recovery site ESXi host via SSH as root.

2. Export the current configuration to a temporary JSON file

   configstorecli config default get -c hbr -g services -k hbrsrv -p vmacore -o /tmp/config.json

3. Open the file using the vi editor
   

   vi /tmp/config.json

4. Locate the thread_pool section and modify max_concurrency to 321 and task_max to 256

   Before:
   

   "thread_pool": {   "io_max": 64,   "io_min": 2,   "max_concurrency": 193,   "task_max": 128,   "task_min": 2,   "thread_name_prefix": "hbrsrv",   "thread_stack_size_kb": 64 }

   After:
   

   "thread_pool": {   "io_max": 64,   "io_min": 2,   "max_concurrency": 321,   "task_max": 256,   "task_min": 2,   "thread_name_prefix": "hbrsrv",   "thread_stack_size_kb": 64 }

5. Save the file and apply the modified configuration
   configstorecli config current set -c hbr -g services -k hbrsrv -p vmacore -j /tmp/config.json
6. Verify the changes were successfully applied
   configstorecli config current get -c hbr -g services -k hbrsrv -p vmacore
7. Restart the hbrsrv service to apply the changes
   /etc/init.d/hbrsrv restart

Additional Information

hbrsrv のデッドロックによりレプリケーションが「ディスクの統合が進行中です」で停止してRPO 違反が発生する