vSAN Witness Traffic Failure Due to vmk0# Network Communication Loss

Products

• VMware vSAN

Issue/Introduction

Symptoms 

• vSAN Witness Traffic Separation is configured on vmk0#.
• vSAN data traffic is configured on vmk#.
• vmkping -I vmk0# <Witness-IP> from all vSAN data nodes results in 100% packet loss.
• vmkping -I vmk0# <Data-Node-IP> from the vSAN Witness Appliance also results in 100% packet loss.
• vSAN Skyline Health reports:

• Ping tests over the management interface (vmk##) from the ESXi hosts to the witness appliance were successful,

[root@esxihost :~ ] vmkping -I vmk## 10.###.#.15
PING 10.###.#.15 (10.###.#.15): 56 data bytes
64 bytes from 10.###.#.15: icmp_seq=0 ttl=61 time=1.001 ms
64 bytes from 10.###.#.15: icmp_seq=1 ttl=61 time=1.058 ms
64 bytes from 10.###.#.15: icmp_seq=2 ttl=61 time=1.019 ms
-- 10.###.#.15 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.001/1.026/1.058 ms

Definitions:

• vmk0#: Dedicated adapter tagged for vSAN witness traffic.
• vmk#: Adapter carrying vSAN data traffic on data nodes.
• vmk##: Adapter carrying Management traffic on data nodes.

Environment

VMware vSAN 7.x

Cause

The issue is caused by a network communication failure on the interface vmk0#, which is dedicated to vSAN witness traffic. As a result, the vSAN data nodes are unable to reach the witness appliance, leading to component isolation.

• This was verified using the following command on each ESXi host:

[root@esxihost :~ ] vmkping -I vmk0# <vSAN witness IP>

PING <vSAN witness IP> (vSAN witness IP): 56 data bytes

sendto () failed (Network is unreachable)

This confirms that vmk0# is not reachable, resulting in witness traffic loss.

Resolution

Engage Network Team to identify and resolve physical or routing issues affecting vmk0#, which is used for vSAN witness communication.