Virtual Machines running on an SEsparse snapshot may report guest data inconsistencies

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Applications such as databases may report block-level data inconsistency.
• Guest operating systems may report file system metadata inconsistencies
• The VM fails to boot when it is running from an SEsparse snapshot.

SEsparse is a snapshot format introduced in vSphere 5.5 for large disks, and is the preferred format for all snapshots in vSphere 6.5 and above with VMFS-6.

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0

Cause

VMware has identified an issue in SEsparse VM snapshots that can cause data inconsistencies.

This issue occurs when a VM is running on an SEsparse snapshot and experiences a burst of non-contiguous write IO in a very short period of time.

Resolution

The issue is resolved the following releases:

• vSphere 6.7 Update 1 - Fore more information, see VMware ESXi 6.7 Update 1 Release Notes.
• vSphere 6.5 Patch - For more information, see VMware ESXi 6.5, Patch Release ESXi650-201810002.
• vSphere 6.0 Patch - For more information, see VMware ESXi 6.0, Patch Release ESXi600-201810001.

Workaround:
This issue can be prevented by disabling “IO coalescing” for SEsparse. 

Note:

• Disabling IO coalescing is accomplished by modifying a configuration option on each host.
• To take effect, VMs must be power-cycled or migrated (vMotion) to other hosts that have the config option set.

Impact of disabling IO coalescing:
Disabling IO coalescing can cause IO performance degradation while VM is running from the snapshot. The extent of degradation depends on the individual virtual machine workload. 

Note: After patches are released, the workaround needs to be rolled back to regain performance benefits of IO coalescing.

The advanced configuration parameters that control IO coalescing are not available through the UI. To make these changes, use the ESXi Shell, PowerCLI, or your preferred API method.

To disable SEsparse IO coalescing using the ESXi Shell:

1. Log into the ESXi host through SSH as root.
2. Determine if IO coalescing is enabled by running this command:
   
   esxcli system settings advanced list -o /COW/COWEnableIOCoalescing. 
   
   Note: The "Int Value" will be "1" if it is enabled.
    
3. If it is enabled, disable IO coalescing by running this command:
   
   esxcli system settings advanced set -i 0 -o /COW/COWEnableIOCoalescing.
    
4. Rerun step 2 to confirm that the "Int Value" has changed to "0"
5. Complete this procedure on all hosts.
6. After all hosts have been configured, enter Maintenance Mode on each host in series, or vMotion all VMs to ensure the new setting takes effect. Alternatively, all VMs can be power-cycled.

To disable SEsparse IO coalescing using PowerCLI:

1. Refer to Using the vSphere PowerCLI utility to manage your vSphere environment for guidance about PowerCLI.

2. Change the advanced configuration parameter on all hosts.

Get-AdvancedSetting -Entity <hostname> -Name "COW.COWEnableIOCoalescing" | Set-AdvancedSetting -Value 0

3. After all hosts have been configured, enter Maintenance Mode on each host in series, or vMotion all VMs to ensure the new setting takes effect. Alternatively, all VMs can be power-cycled.

Additional Information

"File system specific implementation of LookupAndOpen [file] failed" in vmware.log file

Impact/Risks:
Impacted Configurations:

• VMFS-5 or NFS Datastores: VMs with virtual disks >2TB and snapshots. On VMFS-5 and NFS, the SEsparse format is used for virtual disks that are 2 TB or larger
• VMFS-6 Datastores: VMs with snapshots. SEsparse is the default format for all snapshots on VMFS-6 datastores.

Impacted vSphere releases:

• vSphere 6.5 and above with VMFS-6 and any VM with snapshots.
• vSphere 5.5 and above when VMs with virtual disks >2TB have snapshots.

Note: VMFS-5 or NFS datastores only use SEsparse with snapshots when virtual disks exceed 2 TB in size or SEsparse is explicitly configured.