Intermittent Latency observed for VMs located on a vSAN Datastore

Products

• VMware vSAN

Issue/Introduction

• Intermittent Latency can be observed for VMs located on the vSAN Datastore
• Skyline Health may not report any issues
• VMs reporting latency may be hosting applications which are extremely latency sensitive e.g. VMs hosting financial application database with transaction timeout as low as two seconds

Environment

VMware vSAN

Cause

• The issue is caused by intermittent packet drops, packet resets and congestion on the vSAN Network
• VMkernel logs for the master node will report high RT Latency: 

2025-12-24T10:34:47.979Z In(182) vmkernel: cpu45:2100304)CMMDS: LeaderUpdateMeanRTLatency:12423: Throttled: 52fe9c22-0dae-eca9-daf6-######: High RT latency. Node 64f20c66-c849-07c3-04e2-#########, RT latency 818(ms). Mean RT latency 91(ms)

• Native vSAN Traces will report connection resets:

2025-12-24T10:32:53.565057 [19567402] [cpu0] [4332bbba8000] RDTTraceConnectionFailed:2527: {'conn': 0x4332be9beec0, 'addrType': 1, 'transport': 0, 'isClient': False, 'parentType': 'Multiplexer', 'file': 'rdt-iopath.c', 'lineNum': 7263, 'status': 'VMK_ECONNRESET', 'sockErrno': 'VMK_ECONNRESET', 'timeToFailMS': 6661, 'port': 55550}
2025-12-24T10:32:53.565058 [19567403] [cpu0] [4332bbba8000] RDTTraceConnectionHealthChange:2019: {'conn': 0x4332be9beec0, 'addrType': 1, 'transport': 0, 'isClient': False, 'parentType': 'Multiplexer', 'ip': 192.###.##.##, 'oldHealth': 'Healthy', 'newHealth': 'Failed', 'file': 'rdt-iopath.c', 'lineNum': 7263, 'status': 'VMK_ECONNRESET'}

• vSAN Performance charts may report intermittent TCP Congestions:

Resolution

• The issue needs to be looked into by the network team.
• If the vSAN nodes are in the same RACK and the vSAN network connects to the TOR switch, the issue needs to be looked onto by the hardware team