Network latency check intermittently reporting high latency between nodes

Products

• VMware vSAN

Issue/Introduction

Network latency check is triggered in skyline health

Environment

VMware vSAN

Cause

Identify the Master node in the cluster:

vSAN How to find the CMMDS and Stats Master Cluster Master ESXi Host name

The Master node may report latency to other nodes in the cluster:

####-##-##T##:##:##.###Z In(182) vmkernel: cpu2:2098984)CMMDS: LeaderUpdateMeanRTLatency:11978: Throttled: ########-####-####-####-#############: High RT latency. Node <UUID of destination host>, RT latency 1080(ms). Mean RT latency 40(ms)
####-##-##T##:##:##.###Z In(182) vmkernel: cpu75:2098984)CMMDS: LeaderUpdateMeanRTLatency:11978: Throttled: ########-####-####-####-#############: High RT latency. Node <UUID of destination host>, RT latency 1014(ms). Mean RT latency 37(ms)
####-##-##T##:##:##.###Z In(182) vmkernel: cpu10:2098984)CMMDS: LeaderUpdateMeanRTLatency:11978: Throttled: ########-####-####-####-#############: High RT latency. Node <UUID of destination host>, RT latency 891(ms). Mean RT latency 33(ms)

Resolution

Contact your local networking team to investigate causes for network latency between the nodes in the cluster that are reporting issues, e.g this can be caused by CRC errors on the switch.

Please refer to the below documents to assist in this investigation.

Troubleshooting the vSAN Network

vSAN - Physical Network Requirements

Additional Information

vSAN Health Service - Network Health - Network Latency Check

Frequent vSAN "Network latency check" status changes from green to yellow.