High CPU usage on ESX due to full rings / buffers

Products

• VMware vSphere ESXi

Issue/Introduction

High CPU Usage alarm is triggered for ESX and VMs due to buffer exhaustion 

Environment

ESX 8.x

Cause

This is caused by a networking saturation issue which is affecting the Guest operating systems configured for the VMs.  From the hostd log you may see reports of packets being dropped for the target NICs:

####-##-##T##:##:##.###Z Wa(164) Hostd[2103888] [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic2
####-##-##T##:##:##.###Z Wa(164) Hostd[2099918] --> droppedRx: 605463
####-##-##T##:##:##.###Z Wa(164) Hostd[2099918] -->

####-##-##T##:##:##.###Z Wa(164) Hostd[2099922] [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic3
####-##-##T##:##:##.###Z Wa(164) Hostd[2099918] --> droppedRx: 40670
####-##-##T##:##:##.###Z Wa(164) Hostd[2099918] -->

When the stats for the host and VMs are reviewed, full rings and buffers may be reported: 

  client             dvPort  client                 client                                               pktsTx   pktsRx    dropped  dropped  1stRing  1stRing  2ndRing  2ndRing  OutOf
  Name               Id      Type                   SubType         portset       port      pktsTx       M-cast   M-cast    Tx       Rx       Size     Full     Size     Full     Buffers
  ------             ------  ------                 -------         -------       ----      ------       ------   ------    -------  -------  -------  -------  -------  -------  -------
  Management         -       NONE                   NONE            DvsPortset-0  67108872  266447       265540   0         0        0        -        -        -        -        -
  vmname1.eth0       50      VMM Virtual NIC        Vmxnet3 Client  DvsPortset-0  67109282  71035200186  1263412  28428530  2        83541    1024     39312    512      0        141943
  vmname2.eth0       80      VMM Virtual NIC        Vmxnet3 Client  DvsPortset-0  67108893  27154772117  1395175  29281432  0        120419   4096     802      512      235      18403
  vmk0               10      VMKernel TCP/IP stack  NONE            DvsPortset-0  67108881  6075123012   441271   440995    0        0        -        -        -        -        -
  vmk1               18      VMKernel TCP/IP stack  NONE            DvsPortset-0  67108882  2474641904   441040   440969    0        0        -        -        -        -        -

Resolution

To resolve this issue, the ring size and buffers should be reviewed and reconfigured as needed.  For more information on reviewing ESX networking configurations and stats, see the following KB articles: 

The output of esxtop shows dropped receive packets at the virtual switch
Large packet loss in the guest OS using VMXNET3 in ESXi