Enabling or disabling simultaneous write protection provided by VMFS using the multi-writer flag

Products

• VMware vSphere ESXi

Issue/Introduction

VMFS is a clustered file system that disables (by default) multiple virtual machines from opening and writing to the same virtual disk (.vmdk file). This prevents more than one virtual machine from inadvertently accessing the same .vmdk file.

The multi-writer option allows VMFS-backed disks to be shared by multiple virtual machines. This option is used to support VMware fault tolerance, which allows a primary virtual machine and a standby virtual machine to simultaneously access a .vmdk file.

You can use this option to disable the protection for certain cluster-aware applications where the applications ensure that writes originating from two or more different virtual machines does not cause data loss. This document describes methods to set the multi-writer flag for a virtual disk.

Warning: Be careful when you disable this protection. This change might cause data corruption in cases where the applications in the virtual machine do not maintain consistency in the writes performed to the shared disk. As a result, some virtual machine operations and vSphere features are not supported and these are listed below.

Use cases:

• VMware FT relies on VMware vLockstep technology to establish and maintain an active secondary virtual machine that runs in virtual lockstep with the primary virtual machine. The secondary virtual machine resides on a different host and executes exactly the same sequence of virtual (guest) instructions as the primary virtual machine. The multi-writer flag is automatically enabled when you turn on FT in the vSphere Client.

• Third-party cluster-aware applications, in which the applications ensure that writes originate from two or more different virtual machines, does not cause data loss (for example, you would use this for Oracle RAC virtual machines). For this case, manually enable the multi-writer flag using the instructions in this article. Follow the same procedure when using virtual RDMs.

• Starting vSphere 7.0 Update 1, NVMe over Fibre Channel and NVMe over RDMA (RoCE v2) datastores now support multi-writer option enabling in-guest systems that leverage cluster-aware file systems to have distributed write capability. Please refer to the VMware NVMe Storage guidelines before configuring the NVMe Storage for shared disks with the multi-writer attribute. For more information, see About VMware NVMe Storage

• Starting vSphere 7.0 Update 3, vVOL datastore backed by FC, iSCSI protocols now support multi-writer option enabling in-guest systems that leverage cluster-aware file systems to have distributed write capability. For more information, see Oracle workloads on VMware Virtual Volumes (vVOLS) using Pure Storage FlashArray X50 and Broadcom LPe36000 Fibre Channel Adapter – better performance

• Starting vSphere 8.0 Update 2, vVOL datastores backed by NVMe FC, NVMe TCP  also supports multi-writer option .

• For configurations using VMware vSAN, see Using Oracle RAC on vSphere 6.x vSAN Datastore.

• Support information for Veritas InfoScale in ESXi guests can be found at Veritas InfoScale Enterprise 7.x in guest storage, high availability, and disaster recovery support (Partner Verified and Supported)

Supported and unsupported actions or features:
 

Actions or Features	Supported	Unsupported	Notes
Power on, off, restart virtual machine	√
Suspend VM		×
Hot add virtual disks	√		Only to existing adapters
Hot remove devices	√
Hot extend virtual disk	√		For shared disks from vvols backend datastores from vSphere 8.0 update 2 onwards
Connect and disconnect devices	√
Snapshots of VMs with dependent disks		×	Virtual backup solutions leverage snapshots through the vStorage APIs; for example, VMware Data Recovery, vSphere Data Protection. These are also not supported.
Snapshots of VMs with independent-persistent disks	√		Supported in vSphere 5.1 update2 and later versions. The snapshot task will be completed successfully but will skip the independent disks from the snapshot operation. Powered on virtual machines with independent disks cannot support memory snapshots.
Cloning		×
Storage vMotion		×	Neither shared nor non-shared disks can be migrated using Storage vMotion due to the virtual machine stun required to initiate the storage migration.
Changed Block Tracking (CBT)		×
vSphere Flash Read Cache (vFRC)		×	Stale writes can lead to data loss and/or corruption
vMotion	√		Supported for ORAC only and limited to 8 ESX/ESXi hosts

Limitations:

• When using the multi-writer mode for disks from VMFS datastore, the virtual disk must be eager zeroed thick; it cannot be zeroed thick or thin provisioned. For more information, see A virtual machine fails to power on with the error: Thin/TBZ disks cannot be opened in multiwriter mode. VMware ESX cannot open the virtual disk for clustering. (1033570).

• When using the multi-writer mode for disks from vVOLs datastore, the virtual disk can be either thin or thick.

• Hot adding a virtual disk removes Multi-Writer Flag.

• Multiple VMs sharing the same disk with the Multi-Writer Flag enabled will not be allowed to power on when residing on the same host.  This is by design to prevent reservation conflicts on the disk. 

• From vSphere 8.0 update 2 onwards virtual NVMe controller is  also supported for the virtual disks with multi-writer mode.

• From vSphere 6.7 Update 1 onwards, the virtual disks sharing support in multi-writer has been extended to more than 8 hosts. "This applies to VMFS5 / VMFS6 as well". In order to enable this feature, you need to enable /VMFS3/GBLAllowMW advance config option  as shown below:

1. Login to ESXi host and execute the below command:

esxcli system settings advanced set -i 1 -o /VMFS3/GBLAllowMW

Note: This will enable the multi-writer lock. GBLAllowMW config option is disabled by default on ESXi host.
Note: Maximum number of hosts that can support a shared virtual disk is 64 as for vSphere 6.7 Update 1 .

2.  Execute the below command to check the value set.

esxcli system settings advanced list -o /VMFS3/GBLAllowMW
Path: /VMFS3/GBLAllowMW
Type: integer
 Int Value: 1             =========== > Value set to 1 from default 0
Default Int Value: 0
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Allow multi-writer GBLs.

Note: Run esxcli system settings advanced set -i 0 -o /VMFS3/GBLAllowMW to disable this option.

3.  Enable the above config option on ALL the hosts that are part of the shared environment.

Environment

VMware vSphere ESXi 8.0.1
VMware vSphere ESXi 6.7
VMware vSphere ESXi 8.0.2
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.5

Resolution

Enabling Virtual Disk Sharing

To configure a shared virtual disk by setting the multi-writer flag:


For ESXi 6.x

Configure virtual machine & add a second hard disk (thick EZT disk type) by mentioning the sharing as multiwriter. (In virtual machine edit settings > virtual hardware > second hard disk > sharing > Multiwriter)


For previous versions of ESXi
 

1. Create a virtual disk to be shared using any acceptable approach. For example, through the vSphere Client:
   1. Create a new disk for a virtual machine on a VMFS data store:
      
      
       
   2. In the Create a Disk window, for ESX/ESXi versions earlier than 5.0, click Support clustering features such as Fault Tolerance to create the new virtual disk in the required format, eager zeroed thick.
      
      
      
      In ESXi 5.x versions, the Create a Disk window provides Thick Provision Lazy Zeroed, Thick Provision Eager Zeroed, and Thin Provision options under Disk Provisioning. Be sure to select Thick Provision Eager Zeroed. Also, note that the documentation may refer to this as Flat pre-initialized.
      
      
       
   3. Choose a Virtual Device Node setting, such as SCSI (1:0). Make a note of it, as it is required later.
      
      
      
       
2. Add the multi-writer flag to configure sharing.
   
   Note: Do not change the SCSI Controller's SCSI Bus Sharing options. Keep the default selection, None.
   
   To enable Multi Writer flag for sharing a particular disks, apply one of these options:
   1.  
      1. Power off the virtual machine.
      2. In the . vmx file that defines the virtual machine, add an entry similar to:
         scsiX:Y.sharing = "multi-writer"
         
         where X is the controller ID and Y is the disk ID on that controller. The setting screen of a virtual machine shows these values.
         
         
         
         Add this setting for each virtual disk that you want to share. For example, to share four disks, the configuration file entries look like this:
         scsi1:0.sharing = "multi-writer"
         scsi1:1.sharing = "multi-writer"
         scsi1:2.sharing = "multi-writer"
         scsi1:3.sharing = "multi-writer"
          
      3. Save the .vmx file and power on the virtual machine.
         
         Note: In vSphere 6.0 and later, the GUI has the capability of setting Multi Writer flag without any need to edit .vmx file.
          
   2. In the vSphere Client, power off the virtual machine, navigate to Edit Settings > Options > Advanced > General > Configuration Parameters. Add rows for each of the shared disks and set their values to multi-writer.
      
      
       
3. Add this disk to another virtual machine:
    
   1. In the vSphere Client inventory, right-click the virtual machine and click Edit Settings.
   2. Click the Hardware and click Add.
   3. Select Hard Disk and click Next.
   4. Select Use an Existing Virtual Disk.
      
      Note: You do not need to create a disk because the disk is already created.
       
4. Enable sharing on this virtual machine, using one of the approaches listed in step 2.

To configure the multi-writer flag via Host Client:

 

1. Right Click the VM required and then choose Edit Settings.
   
   
    
2. Click on the Tab VM Options.
   
   
    
3. Choose Advanced and then click on Edit Configuration under the Configuration parameters.
   
   
    
4. Add the Settings for each virtual disk that you want to share as below.
   scsi1:0.sharing = ""multi-writer""
   scsi1:1.sharing = ""multi-writer""
   scsi1:2.sharing = ""multi-writer""
   scsi1:3.sharing = ""multi-writer""
   
   
    

NOTE: While migrating the multi -writer-VM, do disconnect the disk and reconnect after migration. Some times you will get duplicate disk information.[ Make sure to use same SCSI-id]. You can follow the process listed in Migrating VMs with shared or multi-writer disks.