During Upgrade from 9.0.1 to 9.1, an alarm indicating "High pNic error rate detected" is Generated on each host that was upgraded - VMware vSphere

Products

• VMware vSphere ESXi

Issue/Introduction

• During the upgrade from 9.0.1 to 9.1, an alarm indicating "High pNic error rate detected" is being generated for each host that was upgraded to 9.1
  
  
• vCenter "/var/log/vmware/vpxd/vpxd.log" logs show that the "Generic error rate" reaches unbelievable levels.
  

  <timestamps> UTC <HOSTNAME> vpxd[6090]: Event [7410167] [1-1] [<timestamps> UTC] [vim.event.EventEx] [error] [vSAN Health] [###] [7410167] [RX Generic error rate reaches 30752‰ on Physical Adapter vmnic2.]
  <timestamps> UTC <HOSTNAME> vpxd[6090]: Event [7410168] [1-1] [<timestamps> UTC] [vim.event.AlarmStatusChangedEvent] [info] [] [###] [7410168] [Alarm 'High pNic error rate detected. Check the host's vSAN performance view for details' on <HOSTNAME> changed from Green to Red]
  <timestamps> UTC <HOSTNAME> vpxd[6090]: Event [7410169] [1-1] [<timestamps> UTC [vim.event.EventEx] [error] [vSAN Health] [###] [7410169] [RX Generic error rate reaches 6186‰ on Physical Adapter vmnic0.]
  <timestamps> UTC <HOSTNAME> vpxd[6090]: Event [7410170] [1-1] [<timestamps> UTC] [vim.event.EventEx] [error] [vSAN Health] [###] [7410170] [RX Generic error rate reaches 9021‰ on Physical Adapter vmnic2.]
  <timestamps> UTC <HOSTNAME> vpxd[6090]: Event [7410171] [1-1] [<timestamps> UTC] [vim.event.EventEx] [error] [vSAN Health] [###] [7410171] [RX Generic error rate reaches 3045‰ on Physical Adapter vmnic0.]
  <timestamps> UTC <HOSTNAME> vpxd[6090]: Event [7410172] [1-1] [<timestamps> UTC] [vim.event.EventEx] [error] [vSAN Health] [###] [7410172] [RX Generic error rate reaches 7524‰ on Physical Adapter vmnic2.]
  <timestamps> UTC<HOSTNAME> vpxd[6090]: Event [7410173] [1-1] [<timestamps> UTC] [vim.event.EventEx] [error] [vSAN Health] [###] [7410173] [RX Generic error rate reaches 6745‰ on Physical Adapter vmnic0.]
  
  

  
  
  
• The ESX host "/var/run/log/hostd.log" shows the incrementing errors:
  

  <timestamps> UTC -WARNING Hostd 2100489 [esx@4413 sub="Statssvc.StatsCollector"] Error stats for pnic: vmnic0
  <timestamps> UTC -WARNING Hostd 2100448 [esx@4413] --> droppedRx: 554
  <timestamps> UTC -WARNING Hostd 2100448 [esx@4413] --> errorsRx: 885024
  --
  <timestamps> UTC -WARNING Hostd 2100489 [esx@4413 sub="Statssvc.StatsCollector"] Error stats for pnic: vmnic2
  <timestamps> UTC -WARNING Hostd 2100448 [esx@4413] --> droppedRx: 552
  <timestamps> UTC -WARNING Hostd 2100448 [esx@4413] --> errorsRx: 282206
  --
  <timestamps> UTC -WARNING Hostd 2100468 [esx@4413 sub="Statssvc.StatsCollector"] Error stats for pnic: vmnic0
  <timestamps> UTC -WARNING Hostd 2100448 [esx@4413] --> droppedRx: 1017
  <timestamps> UTC -WARNING Hostd 2100448 [esx@4413] --> errorsRx: 885024
  --
  <timestamps> UTC -WARNING Hostd 2100477 [esx@4413 sub="Statssvc.StatsCollector"] Error stats for pnic: vmnic2
  <timestamps> UTC -WARNING Hostd 2100448 [esx@4413] --> droppedRx: 608
  <timestamps> UTC -WARNING Hostd 2100448 [esx@4413] --> errorsRx: 282206
  

• The command "esxcli network nic stats get -n vmnic#" shows a large amount of "Total receive errors":
  

   NIC statistics for vmnic2:
        Packets received: 32178817
        Packets sent: 9310136
        Bytes received: 212240029521
        Bytes sent: 1416103020
        Receive packets dropped: 0
        Transmit packets dropped: 0
        Multicast packets received: 87683
        Broadcast packets received: 281835
        Multicast packets sent: 49
        Broadcast packets sent: 97
        Total receive errors: 82624  **************
  

Environment

VMware vCenter 9.1

ESXi 9.0.x and 9.1.x

 

Cause

• The enviornment/VDS being in a "mixed" state (3 hosts on 9.01 and 3 on 9.1) contributed to these alarms/errors.

Resolution

• Complete the upgrade to 9.1.
• After upgrading the DVS and completing the 9.1 upgrade on remaining ESX hosts the error/alarms will no longer generate.