vSAN File Service out of Network after ESXi Host reboot when the Virtual Switch has Promiscuous mode enabled

Products

• VMware vSAN 7.x

Issue/Introduction

After ESXi Host reboot, File Service Health check is in Error state. 
In vSAN Skyline Health, DNS Lookup and Infrastructure Health are in Red/Critical state.
No other changes made in the environment. The Hosts which were not rebooted works fine.

Gateway IP address is not reachable from the container running on File Service VM. This can be validated by running a ping test to the Gateway IP address by attaching to the Container on FSVMs, 100% packet loss is seen. 

Environment

In vCenter GUI, Virtual Switch/portgroup has both "Mac Learning" and "Forged Transmit" in Enabled state.

But when below command is run to check the configuration on ESXi level, the Mac learning shows Disabled on Hosts which were rebooted :

# esxcfg-vswitch -l | grep -i "vsan file service node"

# netdbg vswitch mac-learning port get --dvport <port-id> --dvs-alias <DVS-Name>
MAC Learning:                   False

The Hosts which were not rebooted still has Mac Learning Enabled.

Cause

This can happen if Promiscuous mode was in Enabled state on the Virtual Switch. After rebooting, Mac learning can get disabled on Host level if switch has Promiscuous mode enabled.

Resolution

Enable the Mac Learning on ESXi host using below command :

# netdbg vswitch mac-learning port set --enable --dvport <port-id> --dvs-alias DSwitch_OTA 

Also, disable the Promiscuous mode after consulting with customer.