vSphere Replication is replicating huge data though there are no changes occurring on the virtual machine

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

1. vSphere Replication is currently replicating substantial data despite the absence of changes occurring on the virtual machine.

2. Windows OS reports LSI reset events such as Event ID 129 – LSI_SAS Errors (Reset to device, \Device\RaidPort0, was issued.). 

3. Users observe a decline in performance with Microsoft Outlook, particularly when downloading emails. 

4. Additionally, the guest operating system may become unresponsive or enter a hung state, leading to noticeable slowness in exchange applications. 

It is important to note that when replication is paused for such VMs, none of the aforementioned issues are experienced.

Environment

VMware vSphere Replication 8.x
VMware vSphere Replication 9.x

Cause

This issue arises when UNMAP is enabled in the Guest OS.

UNMAP (or TRIM) is a feature that allows a guest operating system to inform the storage subsystem that certain blocks of data are no longer in use, enabling the storage to reclaim that space.

Resolution

WORKAROUND

How vSphere Replication Works When Using Guest OS Trim/Unmap Commands

If the Guest OS is becoming unresponsiveness or the target datastore (VMFS) is consuming higher space in contrast to the source disk space consumption, you can try these options below - 

1. Disable Guest OS UNMAP in Windows 

A. To verify the current UNMAP setting, launch a command prompt and run the command:
 
fsutil behavior query DisableDeleteNotify
 
DisableDeleteNotify=0 - indicates the Trim and Unmap feature is ON (Enabled)
DisableDeleteNotify=1 - indicates the Trim and Unmap feature is OFF (Disabled)

To enable the feature, run the command:
 
fsutil behavior set DisableDeleteNotify 0 

To disable the feature, run the command:
 
fsutil behavior set DisableDeleteNotify 1
 

B. Disable Guest OS UNMAP in Linux  

To disable in RHEL or other Linux please reference your OS documentation or contact the Linux Distro provider. Some Linux distributions, like Ubuntu and SUSE, automatically perform periodic fstrim operations for you, which can be configured in a weekly cron task.

CAUTION: If you are using thin-provisioned disks, disabling UNMAP can lead to a discrepancy between the size reported by the Guest OS and the actual size of the thin disk as seen by the storage array.

2. Configure ESXi Host Advanced Settings

Configure the DemandlogFailCollidingUnmap option to a value of either 0, 2, or 3. The impact of this setting may vary based on the Guest Operating Systems in use; therefore, it is advisable to apply the configuration that is most appropriate for your specific environment. The changes to the advanced settings will be applied immediately, and a reboot of the host system is not necessary.

---

ESXi Advanced Option "DemandlogFailCollidingUnmap" History

Prior to the release of ESXi version 7.0.3:

The hbr_filter driver managed collisions, which occur during synchronization operations with unmapping from the guest operating system, by initiating on-demand copies of the respective blocks.

Drawback: This method necessitated extra READ and WRITE operations to maintain the overlapping regions, which could result in unforeseen delays.

NOTE: This behavior resembles the subsequently introduced advanced option "DemandlogFailCollidingUnmap," with the selection of option 0.

In ESXi version 7.0.3:

The DemandlogFailCollidingUnmap option was introduced in ESXi 7.0.3.

Default Option: 1

• With the default setting, the hbr_filter driver inhibits SCSI UNMAP commands originating from the guest operating system during a synchronous operation if such commands intersect with the data being transferred to the target datastore.
• The driver conveys a 'busy' status (SCSI event status code 08h) to the guest operating system, with the expectation that the guest OS will subsequently attempt to retry the trim/unmap commands, thereby preventing any adverse effects on applications operating within the virtual machine.

Between ESXi versions 7.0.3 and 7.0.3 P07, as well as 8.0 and 8.0.1

The DemandlogFailCollidingUnmap option was available with two values: 0 and 1.

To verify the current configuration settings on ESXi versions 7.0.3 and later, please execute the following command:

$ esxcli system settings advanced list -o /HBR/DemandlogFailCollidingUnmap

To modify this option, use the command:

$ esxcli system settings advanced set -o /HBR/DemandlogFailCollidingUnmap -i 0

To restore the default setting, you should execute:

$ esxcli system settings advanced set -o /HBR/DemandlogFailCollidingUnmap -i 1
 

For ESXi 7.0.3 P08 or later patches and for ESXi 8.0.2 or later versions

The DemandlogFailCollidingUnmap configuration option has been revised to incorporate two additional values (2 and 3). These new values enhance the flexibility of the drivers response to unmap/trim commands issued by the guest operating system.

New Values in the DemandlogFailCollidingUnmap Configuration Option:

Option value '2': A 'Check Condition' Error (SCSI event status code 02h)

The hbr_filter driver generates a 'check condition' error without providing any additional sense data. As a result, the guest operating system fails the command without attempting an immediate retries.

$ esxcli system settings advanced set -o /HBR/DemandlogFailCollidingUnmap -i 2

Option value '3': 'Success' Response (SCSI event status code 00h)

The hbr_filter driver returns a success status to the trim/unmap command issued by the guest operating system.
 
$ esxcli system settings advanced set -o /HBR/DemandlogFailCollidingUnmap -i 3

Additional Information

Applying storage policy to disk files, the disk consumed space is bigger again. It is storage policy configuration.  Because of the storage policy, the used space is larger than original size. It is expected. For example, if one file is 10GB, after applying RAID 1 policy, it will be about 20GB.

When disabling Guest OS Unmap and the VM is utilizing thin provisioned disks, you will need to consider the following:

• We wont be able to let the array known that blocks have been freed, thus there will be a discrepancy between what vCenter thinks the size of the thin disk is and the size as reported by the GuestOS.
• Best practice for SQL disks is to use EZT disks, not thin provisioned disks