Transitioning to 2-node vSAN Direct Connect during switch maintenance

Products

• VMware vSAN

Issue/Introduction

In VMware vSAN environments, customers may need to maintain cluster availability during physical network switch maintenance or upgrades. This article provides a validated procedure for temporarily transitioning a 2-node vSAN cluster to a direct cross-connect configuration to avoid downtime when the primary switches are offline.

Symptoms:

• Requirement to perform maintenance on Top-of-Rack (ToR) switches carrying vSAN and vMotion traffic.
• Need to maintain vSAN storage availability and cluster quorum during the switch outage.
• Availability of secondary NIC ports or cables for point-to-point host connectivity.

Environment

• VMware vSAN 8.x 
• VMware ESXi  8.x 
• VMware ESX 9.0, 9.1

Cause

The standard vSAN network configuration relies on physical switches. During switch maintenance, a temporary direct-connect topology is required to bridge the data path between the two nodes, preventing a vSAN network partition.

Resolution

To transition to a temporary direct cross-connect without downtime, follow these steps.

Prerequisites:

• Ensure you have a 10GbE or 25GbE cross-connect cable of sufficient length.
• Verify that the vSAN VMkernel adapters on both hosts are in the same subnet.

Procedure:

1. Navigate to the vSphere Client > Networking > vSphere Distributed Switch (vDS).
2. Select the vSAN/vMotion Port Groups > Configure > Teaming and Failover.
3. Identify the secondary uplink intended for the cross-connect (e.g., Uplink #). Move this uplink to the Active position.
4. Move the primary uplink (currently connected to the switch) to the Standby position.
   • Note: Using Standby is preferred over Unused to ensure the stack remains aware of the interface state.
5. Administratively shut down the switch port connected to the primary NIC (e.g., vmnic#) on ####-Host-01.
6. Physically unplug the cable from the host NIC and insert one end of the cross-connect cable.
7. Repeat steps 5 and 6 for ####-Host-02 using the other end of the cross-connect cable.
8. Verify the physical link status:
   • Check that NIC LEDs are green on both hosts.
   • In the vSphere Client, verify the vmnic status shows as Up.
9. Validation (Non-Disruptive):
   • Create a temporary Port Group on the vDS.
   • Create a new VMkernel interface (e.g., vmk#) on both hosts assigned to this Port Group.
   • Assign Uplink # as the Active uplink for this test group.
   • Run a ping test between hosts: 
     
     vmkping -I vmk# <Peer_VMkernel_IP>
     
      
10. Once connectivity is confirmed, update the production vSAN and vMotion Port Groups to use the cross-connect uplink as Active.
11. Run vSAN Skyline Health and verify that vSAN Basic (unicast) connectivity and MTU check return a "Passed" status.

Additional Information

• Ensure MTU settings (e.g., 9000 for Jumbo Frames) are consistent across the direct-connect interfaces.