vSAN Witness connectivity loss and gateway unreachable after Management IP change

Products

• VMware vSphere ESXi

Issue/Introduction

• In a VMware vSAN Stretched Cluster or 2-node configuration, the vSAN Witness Appliance may lose all connectivity and become disconnected from vCenter Server following a change to the Management IP address or VLAN.
• The Witness Appliance shows as Disconnected or Not Responding in vCenter.
• The Witness Appliance is unable to ping its default gateway from the console or DCUI.
• The below error message may be observed in vobd.log:

YYYY-MM-DDTHH:MM:SS.SSSZ In(14) vobd[524611]:  [netCorrelator] 10551946239595us: [vob.net.gateway.set.failed] Cannot connect to the specified gateway ###.##.##.###. Failed to set it.
YYYY-MM-DDTHH:MM:SS.SSSZ In(14) vobd[524611]:  [netCorrelator] 10551946237424us: [esx.problem.net.gateway.set.failed] Cannot connect to the specified gateway ###.##.##.###. Failed to set it.

Environment

VMware vSAN (2-Node or Stretched Cluster)
vSAN Witness Appliance

Cause

This issue is typically caused by one or more of the following environmental or configuration factors:

• VLAN Tagging: Conflict between VLAN tagging configured within the Witness DCUI and tagging on the physical ESXi PortGroup.
• Incorrect Default Gateway: Failure to update the default gateway IP address after changing the Management IP address.
• Appliance PortGroup Misconfiguration: The secondary portgroup on the Witness appliance lacks an active uplink 
• Physical Network/Firewall Drops: Traffic (specifically UDP 12321 and ICMP) is dropped by physical firewalls or misrouted due to incorrect gateway assignments on the data host vmk interfaces.

Resolution

To resolve the connectivity loss, perform the following troubleshooting and configuration steps:

1. Correct VLAN Tagging Conflicts

1. Access the DCUI of the Witness Appliance.
2. Navigate to Configure Management Network > VLAN (optional).
3. Ensure the VLAN ID is set to 0 if tagging is already performed at the portgroup level on the physical ESXi host.
4. Apply changes and restart the management network.

2. Validate Gateway Configuration

1. Access the DCUI of the Witness Appliance.
2. Navigate to Configure Management Network > IPv4 Configuration.
3. Verify the Default Gateway IP is correct. Update if necessary to match the new subnet requirements.

3. Validate Physical Uplinks and PortGroups

1. Log in to the vCenter Server managing the physical host where the Witness Appliance resides.
2. Verify the PortGroup assigned to the Witness Appliance has an Active Uplink and the correct VLAN ID.
3. Ensure the physical switch ports (pNICs) are trunked for the required VLANs.

4. Physical Network/Firewall Drops

1. Involve the Network/Firewall team to verify that traffic is not being dropped by a stateful firewall between the data site and the witness site, specifically for UDP 12321.
2. Ensure UDP 12321 and ICMP traffic are permitted.

Additional Information

Troubleshooting vSAN Witness appliance partitioned from the stretched cluster

How to Change the IP Address of a VMware vSAN Witness Node