Planned network maintenance considerations for vSAN clusters

Products

• VMware vSAN

Issue/Introduction

vSAN is a distributed storage system that combines locally attached disks on each individual physical node to a shared vSAN datastore on selected hosts. The data is then distributed across individual disks on any participating hosts in the vSAN cluster matching your storage policy configuration.

In terms of a vSAN cluster, each individual vSAN node is a client/consumer of the network. vSAN uses the defined vmkernel adapter and the ESXi Network Stack for any vSAN-related network communication.

The network infrastructure is the backbone of a vSAN cluster and the only way for exchanging any kind of data between each individual host. The type of data is, but not limited to:

1. I/Os from virtual machines (intensity depends on I/O workload from VMs and their applied storage policies)
2. All cluster membership updates
3. Periodic node heartbeats between hosts (every second between the primary host and all remaining hosts)
4. Any vSAN-related intra-node management communication, like performance metrics updates.

As the network is the primary method for any essential vSAN-related communication, any brief interruption does have the potential to cause issues in terms of data availability, stability, and/or performance issues on your vSAN cluster.

This can be caused by any potentially disruptive changes to the network infrastructure that handles any vSAN traffic, including but not limited to:

1. Software upgrades of switches/routers
2. Reboots of switches/routers
3. Any potentially disruptive configuration changes on switches/routers (e.g. routing, BGP changes, etc)
4. Any MTU changes on any network equipment or device (including changes on ESXi host, e.g. on vmkernel adapters or virtual switches)
5. Hardware replacements

Resolution

Any potentially disruptive changes to the network infrastructure that is responsible for processing vSAN traffic has the potential to affect the stability and/or performance of vSAN. Therefore, we recommend if you're going to do any sort of network maintenance to the vSAN cluster, do the following:

• Schedule a maintenance window for any maintenance to be performed.
• Use extreme caution when performing any network-related changes to your environment.
• Have a rollback plan present to restore functionality if things don't go smoothly.
• Perform the upgrades/changes one host at a time if possible.
• If your vCenter resides on the vSAN datastore and using a distributed switch set up an ephemeral port for the management network to quickly recover the vCenter. See the below KBs for further info:
  • Static (non-ephemeral) or ephemeral port binding on a vSphere Distributed Switch (1022312)
  • vCenter network connectivity lost - Recover vCenter network when connected to a Distributed Switch (83906)

Additional Information

Potential data interruption while performing Network Maintenance on a vSAN cluster while the Primary/Backup host is in Maintenance Mode. (88359)