RedHat Clustering Suite and SUSE Linux Clustering Applications with shared virtual disks on vSAN

Products

• VMware vSAN
• VMware vSAN 8.x
• VMware Cloud Foundation

Issue/Introduction

Clustering Applications with Shared Virtual Disks on vSAN

vSAN supports high availability (HA) clustering applications running inside virtual machines that use shared virtual disks, enabling multiple VMs (on the same or different ESXi hosts) to simultaneously access a shared virtual disk on a vSAN datastore.

The creation of shareable VMDKs and the enabling of Multi-Writer mode permits the use of in-guest shared-storage clustering solutions, such as Red Hat Clustering Suite or SUSE Linux Enterprise HA.

 

1. Multi-Writer Mode (For Linux HA, Oracle RAC, etc.)

Red Hat Enterprise Linux High Availability Add-On and SUSE Linux Enterprise High Availability Extension clusters are supported on vSAN only when configured using Multi-Writer mode.
Oracle RAC is a popular application that successfully uses this mode as it manages its own locking mechanism.

Important: Multi-Writer mode enables concurrent VM access to a virtual disk by disabling standard storage-level locking. Ensure your guest operating system or application is designed to manage data integrity in this setup

 

• The clustering software running within the guest operating systems is fully responsible for coordinating disk access, enforcing locking semantics, and ensuring data integrity.
  
  
• While multi-writer mode enables concurrent virtual disk access across multiple virtual machines, it lacks the SCSI-3 persistent reservations and storage-level locking mechanisms provided by Physical Bus Sharing. Consequently, the clustering software residing within the guest operating systems assumes full responsibility for coordinating disk access, enforcing locking semantics, and maintaining data integrity.
  
  
• VMware does not perform application-level validation of the various clustering mechanisms implemented within guest operating systems.
  
  
• You must ensure that your application is managing its own locking and reservations; refer to the respective vendor documentation for supported configurations and best practices.

 

2. SCSI-3 Persistent Reservation (SCSI-3 PR) Support (WSFC only)

vSAN's support for SCSI-3 PR -based clustering solutions is limited:

• Only the SCSI-3 PR primitives required for Windows Server Failover Clustering (WSFC) are supported.
• The use of SCSI-3 PR based clustering solutions other than WSFC (including Red Hat and SUSE Linux clustering) is not currently supported on vSAN.
• On vSAN SCSI-3 PR based fencing is supported only for WSFC but not for HA solutions on Linux with Pacemaker (eg : fence_scsi or fence_mpath)

 

Warning: DO NOT enable Multi-Writer mode for any VM/VMDK combination unless the guests are capable of safely arbitrating and coordinating multiple systems accessing the same storage. Enabling Multi-Writer mode for disks that do not use in-guest cluster-aware filesystems may result in data corruption.

With vCenter Server: Use DRS VM-host affinity to distribute VMs across hosts, and offer secondary priority hosts that still try to keep the VMs separate after failover.

Note: Enabling Multi-Writer mode removes support for some virtual machine operations and vSphere features. Refer to this matrix for operation/feature supportability when Multi-Writer mode is enabled:

Supported and unsupported actions or features

Note: For DRS support with Red Hat Clustering see Design Guidance for RHEL High Availability Clusters - VMware Virtual Machines as Cluster Members.

Actions or Features	Supported	Unsupported	Note
Power on, off, restart virtual machine	√
Suspend VM		×
Hot add virtual disks	√		Only to existing controllers that are already attached to the vm.
Hot remove devices	√
Hot extend virtual disk		×
Connect and disconnect devices	√
Snapshots		×	Virtual backup solutions that leverage snapshots are not supported.
Snapshots of VMs with independent-persistent disks	√		Only the shared disks need to be in independent-persistent mode. While Independent-Persistent disk mode is not a hard requirement to enable Multi-Writer option, the default Dependent disk mode would cause the “cannot snapshot shared disk” error when a VM snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.
Cloning		×
Storage vMotion		×
Changed Block Tracking (CBT)		×
vMotion	√		Supported for Red Hat Clustering/SUSE Linux Enterprise HA and limited to 64 ESX/ESXi hosts
Stretch Cluster	√

 

Limitations and Requirements:

• When running a 3rd party Clustering Software like Red Hat Clustering Suite or SUSE Linux Enterprise HA on VMware infrastructure, contact the vendor for any Guest Operating System-level Clustering and Configuration issues.
• Support related to Multi-Writer Configuration and issues will be assisted by the VMware Technical Support team.
• To configure the Multi-Writer flag for Enabling Virtual Disk Sharing, see Enabling or disabling simultaneous write protection provided by VMFS using the Multi-Writer flag
• RHEL 6.x and higher, or SUSE Linux 11 and higher, are the supported versions with Multi-Writer mode.
• As VMware vSAN does not support Raw Device Mappings (RDMs), this article applies only to virtual disks resident on the vSAN datastore.
• Starting in vSAN 6.7 Patch 01, EZT disk is no longer required for the Multi-Writer mode to be enabled on vSAN.
• Note: Maximum number of hosts that can support a shared virtual disk is 64.
  
  

Environment

VMware vSAN 8
VMware vSAN 9

Resolution

Overview

The process of configuring a Red Hat cluster or SUSE Linux Enterprise HA on a vSAN datastore needs to be done once per cluster at creation time. This requires these steps:

• Creating a VM Storage Policy that will be applied to the virtual disks that will be used as the cluster’s shared storage.
• Adding a storage controller to the VM.
  
  
• Creating the shared virtual disks in eager-zeroed mode with Multi-Writer mode.
  
  
• Attaching the shared disks to one or more virtual machines with Multi-Writer mode.
  
  
• Applying the virtual machine Storage Policy to the shared disks.

The steps illustrated below use Red Hat cluster as an example. SUSE Linux Enterprise HA configuration would require similar steps.

Create a VM Storage Policy

Depending on your Virtual Machine design specifications, you will need to define the VM Storage Policy that will be applied to the Red Hat cluster shared disks. Create a storage policy similar to this figure:
 

Note:

In this example, the VM Storage Policy is named as RHCS

• • Object Space Reservation: 100% which pre-allocates all object’s components on disk.
  • Number of failures to tolerate: 1 is the default which will provide a mirror replica on another vSAN node. This provides RAID 1 protection against one host/hardware component failure. As the dialog shows, this will consume double the storage space on the vSAN datastore.
  • Number of disk stripes per object: The default vSAN policy is 1. For this policy select the desired stripe width. This is similar to RAID 0. In this example, we are using the value of 2.

For more information about storage policy configuration options, see the VMware vSAN documentation.

 

 

Add a storage controller to the Red Hat Enterprise Linux virtual machines

Note: Create controllers of the same type and in the same position (SCSI address) on each Red Hat Enterprise Linux virtual machine.

1. 1. Right-click the VM in the vSphere Web Client and select Edit Settings.
   2. At the bottom of the subsequent window, select SCSI Controller from the New device: drop-down menu and then click Add.
   3. Expand the New SCSI Controller entry.
   4. Leave the Bus Sharing mode as None, and choose the desired type (LSI Parallel or LSI SAS or Paravirtual).
      
      Note: The Paravirtual controller type is generally recommended for Red Hat cluster shared disks.
      
      
   5. Click OK.
   6. Repeat this process on the remaining Red Hat Enterprise Linux virtual machines.

 

 

 

Creating shared disks on the first virtual machine

In vSphere version 6.5 and onwards, the vSphere Web Client has the option to create eager-zeroed thick (EZT) disks on the vSAN datastore.

Note: The virtual disks should be added to the same SCSI positions on each virtual machine. If a disk is in position 1:0 on one virtual machine, it should be in position 1:0 on all virtual machines in the Linux Cluster.

Using the vSphere Web Client

To create shared disks on the first virtual machine using the vSphere Web Client:

1. 1. Right-click the appropriate virtual machine and select Edit Settings.
   2. Select New Hard Disk from the New Device drop-down menu and then click Add.
   3. Expand the New Hard Disk entry and select the Thick provision eager zeroed option in the Disk Provisioning drop-down menu.
   4. In the Sharing drop-down menu, select the Multi-Writer option.
   5. Change the Disk Mode to Independent-Persistent. 
   6. Modify the Virtual Device Node, as appropriate.
   7. Click OK to save the changes.
   8. Starting in vSAN 6.7 Patch 01, EZT disk is no longer required for the Multi-Writer mode to be enabled on vSAN.
   9. Repeat this process for the remaining shared disks.
      
      

 

 

 

 

Adding shared disks to one or more virtual machines

To add shared disks to one or more virtual machines using the vSphere Web Client:

1. 1. Right-click the appropriate virtual machine and select Edit Settings.
   2. Select Existing Hard Disk from the New Device drop-down menu and then click Add.
   3. Navigate to the applicable directory and select the disk.
   4. Click OK.
   5. Expand the New Hard Disk entry and modify the Virtual Device Node, as appropriate.
   6. In the Sharing dropdown menu, select the Multi-writer option.
   7. Change the Disk Mode to Independent-Persistent.
   8. Click OK to save the changes.
   9. Repeat this process for the remaining virtual machines and shared disks.
      
      

 

 

Applying the VM Storage Policies to the shared disks

After the shared disks are created and added to all virtual machines using one of the three methods mentioned above, you must apply the storage policy created for the Red Hat cluster shared disks. The policy must be applied to all applicable disks on all Red Hat Enterprise Linux virtual machines.

1. 1. In vSphere Web Client, right-click the virtual machine and click Edit Settings.
   2. Expand the appropriate hard disk entry.
   3. Choose the desired VM storage policy from the dropdown list.
   4. Verify that the disk mode is Independent – Persistent.
   5. Click OK.
      
      
   6. Repeat this process for all applicable virtual machines and disks.
      
      Note: If you want to apply a different storage policy to the Red Hat cluster shared disks later, the policy change must be executed against all shared disks and on all virtual machines sharing that disk.

 

 

Additional Information

For more information on configuring and managing Red Hat cluster or SUSE Linux Enterprise HA, see:

• Red Hat Enterprise Linux 9 High Availability Add-on Administration
• SUSE Linux Enterprise High Availability Documentation