High RDT Checksum Mismatch count detected

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• ESXi/vCenter 8.0 or higher
• High RDT Checksum Mismatch count alerts
• You see the below messages in vmkernel and vmkwarning logs, where the header IP mentioned is a network scanner

2023-09-18T17:14:41.185Z Wa(180) vmkwarning: cpu40:2116951)WARNING: RDTTCPConn: RDTTCPReceive:3539: 0x43204475ab40(0x0): unexpected RDT >header from IP 10.xx.xx.40 to IP 192.xxx.xx.15 (vmknic vmk0): version 22 type 3 length 454 optionLen 1 ctr: 45
2023-09-18T18:03:59.206Z Wa(180) vmkwarning: cpu39:2116951)WARNING: RDTTCPConn: RDTTCPReceive:3539: 0x43204475ab40(0x0): unexpected RDT >header from IP 10.xx.xx.40 to IP 192.xxx.xx.15 (vmknic vmk0): version 22 type 3 length 454 optionLen 1 ctr: 54
2023-09-24T21:31:45.506Z Wa(180) vmkwarning: cpu39:2116951)WARNING: RDTTCPConn: RDTTCPReceive:3539: 0x43204475ab40(0x0): unexpected RDT >header from IP 10.xx.xx.40 to IP 192.xxx.xx.15 (vmknic vmk0): version 22 type 3 length 454 optionLen 1 ctr: 63

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSAN 8.0.x

Cause

A new health check "checksumMismatchCount" was introduced in vSAN 8.x which may be triggered when RDT TCP port 2233 receives non-RDT traffic.

The threshold is:
- yellow: 1 checksumMismatchCount in 5 minutes
- red: 10 checksumMismatchCount in 5 minutes

If network scanners are in use and polling on RDT port 2233, this introduces non-RDT traffic and thus triggering this alert.

 

As best practice vSAN requires users to create a dedicated VLAN, to separate vSAN from other network traffic

Every time a connection is made to one of the vSAN ports, vSAN checks to see if the RDT header is correct, and would handle it only once confirmed true.
In case this is not confirmed to be correct a connection from such a port scanner will be refused at the application level because it lacks the correct RDT header.

Some system resources will be used in this activity, and these non-vSAN connections are not anticipated to occur.

While it is happening depending on how many unexpected packets are injected this might cause RDT network latency and impact vSAN performance, as vSAN has no DoS protection mechanism.

Resolution

• Remove RDT port 2233 from the network scanner tasks.
• If there are no network scanners in the environment open a case with vSAN support for further investigation.

Additional Information

VMware Ports and Protocols: vSAN
vSAN Networking Terms and Definitions