Intermittent network connectivity loss due to high loads from security scanning

Products

• VMware vSphere ESXi

Issue/Introduction

• Multiple ESXi hosts are experiencing intermittent network connectivity issues. The problem temporarily resolves but recurs on other hosts within the same environment.
• Those ESXi hosts are connected to the same cascading switch.
• Using esxtop (press n for networking view), RX traffic for VMs is low, while RX traffic on vmnic interfaces is unusually high..
• No drops observed via esxcli network nic stats get -n vmnic# .

Environment

VMware vSphere ESXi 

Cause

The issue is not caused by ESXi.

• A security scanner is generating excessive traffic, overwhelming the physical switch.
• This leads to packet drops at the physical switch layer, causing intermittent connectivity issues for ESXi hosts.

Resolution

1. Use the pktcap-uw tool on ESXi to capture traffic on the affected vmnic interfaces via Packet capture on ESXi using the pktcap-uw tool.
2. Check the physical switch interface statistics for drops, discards, or errors.
3. Engage physical switch vendor to troubleshoot further.