Simulating vSphere High Availability failover

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article describes several methods to simulate VMware High Availability (HA) failover in your environment for cluster testing purposes.

Note: This article assumes that you are working with an HA enabled cluster in vCenter Server consisting of 2 ESXi/ ESX hosts, where the Management Network is uplinked in a redundant vmnic configuration.

 

Caution: VMware recommends you perform this simulation only in a Testing or Development environment. DO NOT perform this simulation in a Production environment.

Environment

VMware vSphere ESXi 5.5 and later.

Resolution

You can simulate HA failover depending on the version of vSphere deployed in the environment.

Select a method below depending on your installed version of vSphere.

Method 1

In vSphere 5.x and later., HA is provided by the Fault Domain Manager (FDM) agent deployed on each of the HA cluster hosts. FDM is used where both Network and Datastore Heartbeats are used to determine the availability of a host, and in determining types host failure, whether that is a physically failed host or a Network Isolation type of failure. The FDM agent on secondary hosts report uptime information to the primary host's FDM agent. The primary host communicates with vCenter Server to report the uptime of itself and all secondary hosts.

For example, there are two ESXi/ ESX hosts with two vmnics in a redundant NIC team serving Management Network traffic. These hosts are also sharing a single shared datastore. You want the virtual machines to failover to the surviving host in the cluster.

To prepare the environment for failover simulation:
 

1. Log in to the vCenter Server with the vSphere Client.
2. Edit the Cluster Settings.
3. Under vSphere HA settings, change the Datastore heartbeat to None. Ensure no datastores are selected from the available list, and select Select only from my preferred datastores.
   
   To disrupt the communication between a single host and the vCenter Server, you can physically disconnect the patch cable where these physical NICs are uplinked. Alternatively, you may issue a command to your switch software to disconnect the ports.
   
   When the network communication between the host and the primary host is disrupted (or the primary host and vCenter Server if this host is the primary node) is disrupted, vCenter Server waits for the timeout period where it does not receive communication from the host it is managing, and then declares the host as Isolated. This causes all virtual machines to register and restart on the surviving host.

Method 2

As mentioned in Method 1, disconnecting the network to forcibly disrupt communication between primary and secondary hosts is an option in simulating HA failover. However, to simulate a power-outage or hardware fault type of failure, hard power off the host physically or by using a remote management application such as KVM, DRAC, iLO, or RAS.
 

Method 3

Note: Use of this method may require re-installation of ESXi/ ESX if the kernel module is not properly disabled/re-enabled. When disabling the kernel module for the physical NIC, you lose all remote management through the ESXi Service Console, and can only remotely manage the host through KVM, DRAC, iLO, or RAS. Be sure to have physical access to the host if a remote management application is not available.

Method 3 allows one to simulate a network isolation again, but this time by disabling the physical NIC (vmnic) driver module from the VMkernel, instead of physically disconnecting a patch cable or interrupting connectivity at the physical switch layer.

First determine which module is in use by the physical NIC by using below article.

Determining Network/Storage firmware and driver version in ESXi

Next, disable the module by running the command:

• For vSphere 5.x and later: esxcli system module set --disabled module_name