vSAN cluster alarms and unexpected virtual machine restarts due to physical network packet duplication

Products

• VMware vSphere ESXi

Issue/Introduction

• The vSAN cluster generates network-related alarms, and virtual machines restart unexpectedly.
• Jumbo Frame (MTU) communication fails despite all vSAN nodes residing on the same Layer 2 network.
• Executing vmkping between vSAN VMkernel interfaces (e.g., from source vmk# to destination vmk#) yields no responses, indicating a network communication failure.

Environment

VMware vSAN

VMware vSphere ESXi

Cause

An external interface on the physical network is duplicating traffic, causing network instability and vSAN communication drops. Packet captures performed on the destination host during ICMP testing reveal duplicated requests and replies associated with a MAC address that does not belong to the affected vSAN cluster.

Resolution

1. Perform a packet capture on the affected ESXi hosts while executing vmkping tests to identify duplicated ICMP packets.

2. Review the packet capture to identify the external MAC address responsible for the duplicated requests and replies.

3. Work with the network administration team to trace the identified MAC address to its corresponding physical switch port.

4. Administratively shut down the physical switch port aligned to the offending MAC address to stop the packet duplication.

5. Re-test network connectivity using vmkping to confirm normal traffic flow without duplicated ICMP requests.

6. Verify that all packet loss has resolved and vSAN Skyline alarms have cleared in the vSphere Client.

Additional Information

For more information on capturing and testing of network traffic on ESXi, refer to Broadcom documentation regarding the pktcap-uw and vmkping utilities.

Packet capture on ESXi using the pktcap-uw tool

Testing VMkernel network connectivity with the vmkping command