VM Performance issue noticed during high IO load

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• VMs Performance is too slow.

• VM shows blank page on web console.

• Copying files between the virtual machines and migration(vmotion) reported to be slow.

• Noticed high read/write latency under VSAN performance [Cluster > Monitor > vSAN > Performance]

• Alert "High pNic error rate detected" noticed on the ESXi summary page.

Verification:

• Verify that all vSAN health check categories display healthy and green status on the cluster.

• CRC errors are noticed under var/run/log/hostd.log.

DateTxx:xx:xx.xxxx Wa(164) Hostd[2102792]: [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic#
DateTxx:xx:xx.xxxx Wa(164) Hostd[2102782]: -- > errorsRx: 178317
DateTxx:xx:xx.xxxx Wa(164) Hostd[2102782]: -- > RxLengthErrors: 2001
DateTxx:xx:xx.xxxx Wa(164) Hostd[2102782]: -- > RxCRCErrors: 88158
DateTxx:xx:xx.xxxx Wa(164) Hostd[2102782]: -- >
DateTxx:xx:xx.xxxx Wa(164)) Hostd[2102792]: [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic#
DateTxx:xx:xx.xxxx Wa(164)) Hostd[2102782]: -- > errorsRx: 178313
DateTxx:xx:xx.xxxx Wa(164) Hostd[2102782]: -- > RxLengthErrors: 2001
DateTxx:xx:xx.xxxx Wa(164) Hostd[2102782]: -- > RxCRCErrors: 88156
DateTxx:xx:xx.xxxx Wa(164) Hostd[2102782]: -- >

 

• CRC errors are noticed on nic stats:
  
     Run the follow command against the vmnic which is used for vSAN uplink.

• • $ esxcli network nic stats get -n <vmnic#>
NIC statistics for vnnic1
Packets received: 5721419054
Packets sent: 6897046642
Bytes received: 2845905140057
Bytes sent: 4960832527165
Receive packets dropped: 0
Transmit packets dropped: 0
Multicast packets received: 133976174
Broadcast packets received: 49838376
Multicast packets sent: 390456
Broadcast packets sent: 39197
Total receive errors: 4028
Receive length errors: 36
Receive over errors: 0
Receive CRC errors: 1996
Receive frame errors: 0

• Watch command help in actively refresh the page and from this command one can monitor is the CRC stats are changing.

$ watch esxcli network nic stats get -n <vmnic#>

Environment

VMware ESXi Version: 7.x
VMware ESXi Version: 8.x

VMware vSAN : 7.x
VMware vSAN : 8.x

Cause

• The CRC (Cyclic Redundancy Check) error occurs when data corruption is detected during transmission over a physical network. It happens when the calculated checksum of the received data does not match the expected value, indicating possible data corruption due to issues like faulty cables, SFP, Network Interface Card (NIC) or SAN Switch.

Resolution

Engage the storage and switch team/vendor to further investigate the issue.