hostd is not responding when running test recovery or planned migration by SRM and vSphere Replication for over 10 virtual machines

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• hostd is not responding from vCenter Server when running test recovery or planned migration by SRM and vSphere Replication for over 10 virtual machines
• Guest OS quiescing setting is enabled.
• Test recovery or planned migration failed in step 1: Synchronize storage due to:

 

VR synchronization failed for VRM group: testVM_XX. A generic error occurred in the vSphere Replication Management Server. Exception details: 'java.net.SocketTimeoutException: Read timed out'.

Environment

VMware vCenter Site Recovery Manager 8.x
VMware vSphere Replication 8.x
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere Replication 6.1.x
VMware vSphere ESXi 6.7
VMware Site Recovery Manager 6.1.x

Cause

In test recovery or planned migration, SRM triggers virtual machine's replication for protected virtual machines with creating (quiesed) snapshots at a same time.
Hostd service might not manage to handle too many tasks at a time.

Resolution

This is a known issue.

 

 

Workaround:
To workaround the issue, reduce a number of concurrent snapshots not more than 8:

1. Back up /etc/vmware/hostd/hbrsvc.xml.

# cp /etc/vmware/hostd/hbrsvc.xml /etc/vmware/hostd/hbrsvc.xml.backup

2. Add the following setting under <ConfigRoot> in /etc/vmware/hostd/hbrsvc.xml:

<hbrMaxConcurrentSnapshotOperations>8</hbrMaxConcurrentSnapshotOperations>

3. Save /etc/vmware/hostd/hbrsvc.xml.
4. Restart hostd service (e.g. /etc/init.d/hostd restart).

NOTE: The parameter depends on the environment work load etc..you might need to set the parameter less than 8.  

Additional Information

This issue was fixed in following releases. The fix reduces the maximum number of concurrent snapshots to 8.

• ESXi 6.0 P08
• ESXi 6.5 Update 3
• ESXi 6.7 Update 3

This issue is not affected in ESXi 7.0 or later.