Replication failures occur in VMware Database Management Essentials 1.0

Products

• VMware Data Services Manager

Issue/Introduction

Symptoms:

• An initial replication or full synchronization fails to complete.
• Deltas fail to complete and DiskQueue is full errors may appear in the hbsrv.log from the VMware Database Management Essentials Snapshot Manager.

Environment

VMware Database Management Essentials 1.x

Cause

There are known issues currently affecting replication for VMware Database Management Essentials 1.0 on vSphere 6.7 GA and 6.7 U1.

First, there can be an issue where the replication appears to be in progress, but the checksum fails to complete. The replication fails to resolve until the host is rebooted.

Additionally, it is possible that more demand log chunks be scheduled for parallel transfer than the actual number that can be transmitted. If you are replicating a VM that is running on such a host and this coincides with a slow target host or temporary network errors, this might result in replication failure and DiskQueue is full errors.

 

Resolution

These Issues are resolved in vSphere 6.7 U2 and above.

Workaround:

If the environment is encountering either of these issues, but cannot be updated to vSphere 6.7 Update 2 or above, the following steps can be used to work around the behavior and need to be applied to each affected ESXi host:

1. Migrate all VMs and power off the ones that cannot be migrated on the ESXi host.
2. Edit the advanced settings on the ESXi host and adjust the following items:
   1. Set HBR.ChecksumUseChecksumInfo to 0
   2. Set HBR.DemandlogTransferMaxNetwork to 63
3. Place the host in maintenance mode.
4. Reboot the ESXi host.
5. Confirm that the host is healthy, then remove the host from maintenance mode and reintroduce the workloads, as appropriate.

Please note, adjusting the checksum functionality can lead to the transmission of blocks that have not changed and it would be recommended to revert these changes after updating to a version that addresses the issues.

Additional Information

For additional information regarding the configuration of the advanced settings on an ESXi host, please refer to the following:

Configuring advanced options for ESXi/ESX (1038578)