vSAN host is down after increasing ring buffer size

Products

• VMware vSphere ESXi

Issue/Introduction

• After performing the below steps to update vmnic ring buffer size, the ESX host began experiencing sporadic connectivity issues with vCenter and other ESX hosts
  • Place host into Maintenance mode
    • Down nic: esxcli network nic down -n vmnic#
    • Update vmnic ring buffer: esxcli network nic ring current set -n vmnic# -r 8192 -t 8192
• vSAN health checks were showing alarms for "basic (unicast) connectivity check" and "mtu check"
  
  
• • vSAN VMK is configured to use vmnic3 (uplink1) by default/Active and vmnic2 (uplink2) as standby.

Environment

VMware vSphere ESXi

Cause

• Attempting to ping the between the hosts specified in the health check reveal ICMP failures.
  • vmkping -I vmk# #.#.#.# -s <mtu> -d -c 2000 -i 0.005 | grep -C 5 packet
• The following command revealed ARP was incomplete.
  
  • esxcli network ip neighbor list
• Because MAC was learned on multiple switchports, the return/reply ARP's were not being sent to the "active" nic. 

Resolution

• Investigate with physical network team why the MAC is not updating across switchports. 

Additional Information

• Broadcom BCM5720 (ntg3 Driver) NIC Reports False Link-Up Status to ESXi
• vmnic disconnected in vCenter ui but up at hardware level.