Similar to VMFS and NFS datastores, VMware vSAN prevents multiple virtual machines from opening the same virtual disk (VMDK) in read-write mode. This is to protect the data stored on the virtual disk from corruption caused by multiple writers on the non-cluster-aware filesystems used by most guest operating systems.
To enable in-guest systems that leverage cluster-aware filesystems that have distributed write (multi-writer) capability, you must explicitly enable multi-writer support for all applicable virtual machines and VMDKs.
Actions or Features |
Supported |
Unsupported |
Notes |
Power on, off, restart virtual machine |
√ | ||
Suspend VM |
× | ||
Hot add virtual disks |
√ |
Only to existing adapters | |
Hot remove devices |
√ | ||
Hot extend virtual disk |
× | ||
Connect and disconnect devices |
√ | ||
Snapshots |
× |
Virtual backup solutions leverage snapshots through the vStorage APIs; for example, VMware Data Recovery, vSphere Data Protection. These are also not supported. | |
Snapshots of VMs with independent-persistent disks |
√ |
Only the shared disks need to be in independent-persistent mode. While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the “cannot snapshot shared disk” error when a VM snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software. | |
Cloning |
× | ||
Storage vMotion |
× | ||
Changed Block Tracking (CBT) |
× | ||
vSphere Flash Read Cache (vFRC) |
× |
Stale writes can lead to data loss and/or corruption | |
vMotion |
√ |
|
Supported for Red Hat Clustering/SUSE Linux Enterprise HA and limited to 8 ESX/ESXi hosts |
Stretch Cluster | √ |
Virtual Use case:
Limitations and Requirements:
Note:
The eight hosts limit applies to multi-writer mode shared virtual disks only. Otherwise, the vSphere cluster size can be up to the maximum size supported by the license in use. However, from vSphere 6.7 Update 1 onwards, the virtual disk sharing support in multi-writer has been extended to more than 8 hosts. This applies to VMFS5 / VMFS6 as well. In order to enable this feature, you need to enable /VMFS3/GBLAllowMW advanced config option as shown below:
Log in to ESXi host and execute the below command:
#esxcli system settings advanced set -i 1 -o /VMFS3/GBLAllowMW
In a shared environment such as Raw Device Mapping (RDM) disks, you cannot use Multi-Writer Locks for virtual machines on more than eight ESXi hosts. If you migrate a virtual machine to a ninth host, it might fail to power on with the error message Could not open xxxx.vmdk or one of the snapshots it depends on. (Too many users). This fix makes the advanced configuration option /VMFS3/GBLAllowMW visible. You can manually enable or disable multi-writer locks for more than 8 hosts by using generation-based locking. This issue is resolved in this release. For more details, see VMware knowledge base article 1034165.
Enabling or disabling simultaneous write protection provided by VMFS using the multi-writer flag (1034165): https://kb.vmware.com/s/article/1034165
Overview
The process of configuring a Red Hat cluster or SUSE Linux Enterprise HA on a vSAN datastore needs to be done once per cluster at creation time. This requires these steps:
In this example, the VM Storage Policy is named as RHCS
For more information about storage policy configuration options, see the VMware vSAN documentation.
Add a storage controller to the Red Hat Enterprise Linux virtual machines
Note: Create controllers of the same type and in the same position (SCSI address) on each Red Hat Enterprise Linux virtual machine.
For vSphere 6.5 or newer
Creating shared disks on the first virtual machine
Starting with vSphere version 6.5, the vSphere Web Client has the option to create eager-zeroed thick (EZT) disks on the vSAN datastore.
Note: The virtual disks should be added to the same SCSI positions on each virtual machine. If a disk is in position 1:0 on one virtual machine, it should be in position 1:0 on all virtual machines in the Linux Cluster.
Using the vSphere Web Client
To create shared disks on the first virtual machine using the vSphere Web Client:
Adding shared disks to one or more virtual machines
To add shared disks to one or more virtual machines using the vSphere Web Client:
For vSphere 6.0 Update 1 or newer, but below version 6.5
Creating Eager-Zeroed Disks (PowerCLI)
In vSphere version 6.0 Update 1 or later versions, (not later than version 6.5, Example: 6.0 Update 3), the vSphere Web Client cannot create eager-zeroed disks on the vSAN datastore. While the vSAN Datastore supports eager-zeroed disks, this functionality is not exposed in the vSphere Web Client. To accommodate this limitation in the current release, we must use either PowerCLI or the ESXi command line to create the eager-zeroed disks. This section outlines the PowerCLI method to create disks, and is the preferred method.
Creating Eager-Zeroed Disks (ESXi command line)
In vSphere version 6.0 Update 1 or later, (not later than version 6.5, Example: 6.0 Update 3), the vSphere Web Client cannot create eager-zeroed disks on the vSAN datastore. While the vSAN Datastore supports eager-zeroed disks, this functionality is not currently exposed in the vSphere Web Client. To accommodate this limitation in the current release, we must use either PowerCLI or the ESXi command line to create the eager-zeroed disks. This section outlines the ESXi command line method to create disks.
Note: You must enable the local ESXi shell or SSH access to the host and log in as privileged (root) user to complete this procedure.
Adding shared disks to one or more virtual machines
After the eager-zeroed disks are created, you must add them to the remaining Red Hat Enterprise Linux virtual machines using either the vSphere Web Client or PowerCLI.
Note: The virtual disks should be added to the same SCSI positions on each virtual machine. If a disk is in position 1:0 on one virtual machine, it should be in position 1:0 on all virtual machines in the Red Hat cluster.
Using the vSphere Web Client
To add shared disks to one or more virtual machines using the vSphere Web Client:
Using PowerCLI
For vSphere version prior to 6.0 Update 1
Enabling multi-writer mode for the VMs and Disks using the ESXi shell
The addition of the multi-writer flag is not possible via the vSphere Web Client version prior to 6.0 Update 1. If vSphere 6.0 Update 1 or newer is not installed, use the ESXi shell to enable multi-writer mode on the applicable virtual machines and disks.
Note: As this involves modifying and loading virtual machine configurations, it is recommended that all Red Hat Enterprise Linux virtual machines be registered to the same ESXi host for the purposes of making these changes so we do not have to log in to multiple hosts. The VMs can be distributed throughout the vSphere cluster after making these changes.
Applying the VM Storage Policies to the shared disks
After the shared disks are created and added to all virtual machines using one of the three methods mentioned above, you must apply the storage policy created for the Red Hat cluster shared disks. The policy must be applied to all applicable disks on all Red Hat Enterprise Linux virtual machines.
For more information on configuring and managing Red Hat cluster or SUSE Linux Enterprise HA, see: