"High pNic error rate detected" Alarm During VMware Cloud Foundation 9.1 Upgrade

Products

• VMware vSphere ESXi
• VMware Cloud Foundation

Issue/Introduction

• During an upgrade to VMware Cloud Foundation (VCF) 9.1, a "High pNic error rate detected" alarm is triggered.
• This behavior only occurs after the upgrade has been initiated but before all ESXi hosts have been successfully upgraded.
• When inspecting the ESXi host directly, the actual Rx errors (e.g., missed or length errors) are recorded as zero (or are very low).

Environment

VMware Cloud Foundation 9.1

Cause

An alarm is triggered during the transitional phase of the VCF 9.1 upgrade before all hosts are operating on the target version.

Resolution

1. Verify that the actual Rx errors on the affected ESXi host are very low (or zero) to confirm this alarm is due to this issue.
   • From an SSH session to a host mentioned in the alarm, run the below command (replacing <vmnic#> with the actual vmnic) for each vmnic in the alarm:

     esxcli network nic stats get -n <vmnic#> | grep errors

   • Example command and output for vmnic0:

     [root@<hostname>:~] esxcli network nic stats get -n vmnic0 | grep errors
        Total receive errors: 0
        Receive length errors: 0
        Receive over errors: 0
        Receive CRC errors: 0
        Receive frame errors: 0
        Receive FIFO errors: 0
        Receive missed errors: 0
        Total transmit errors: 0
        Transmit aborted errors: 0
        Transmit carrier errors: 0
        Transmit FIFO errors: 0
        Transmit heartbeat errors: 0
        Transmit window errors: 0

2. Ignore the warnings and alarms if they only began appearing after the VCF 9.1 upgrade was started.
3. Complete the VCF upgrade across all remaining hosts; the alarms should cease once the environment is fully upgraded.

Additional Information

VMware by Broadcom Engineering is aware of and is investigating the exact cause of this issue.

If the alarms were present in the environment prior to initiating the VCF 9.1 upgrade -- or after the VCF 9.1 upgrade is fully complete, they are likely real issues. In that scenario, please consult "High pNic error rate detected" alarm triggered for vSAN ESXi host for further troubleshooting.