vSAN performance insights reports: "Network Congestion Detected During Resync Activity”

Products

• VCF Operations
• VMware vSAN

Issue/Introduction

This article explains the vSAN performance insights issue: "Network Congestion Detected During Resync Activity", why it might be showing up, and what possible solutions there are to address the issue.

When vSAN ESA is running in environments with 10G networking infrastructure, under certain conditions resynchronization traffic (due to maintenance mode, capacity rebalance, policy change, or fault recovery) may cause an impact to VM IO traffic beyond the 20% target that Adaptive Resync attempts to enforce. Guest latency may increase as a result.

Symptoms:

• High VM disk latency using vSAN during resync operations such as rebuilding after HW failure, node removal, disk/disk group removal, or rebalancing.
• High vSAN network throughput activity

Environment

VMware vSAN ESA 8.2 and above

 

Cause

• When 10G networking is used in conjunction with certain high throughput workload configurations, it is possible that specific parameters of the bandwidth sharing algorithm may not provide the same traffic sharing guarantees seen on higher throughput networks.
• This affects all versions of ESA when configured on 10G networks. Support for 10G networking was only added in 80u2. There is no currently released version which contains an automated fix, so the below workaround is recommended.

Resolution

We recommend setting the following option for every system using a 10Gb network for vSAN ESA.  Resync can happen at any time due to failures or automatic data rebalance.
Note: If vSAN is running over 25G or faster, setting the option is NOT recommended because it will make resync data movement take longer than needed on these networks.

Setting a specific advanced configuration option can restore the scheduler's ability to balance resynchronization traffic and VM traffic fairly on 10G networks. The option /VSAN/DOMNetworkSchedulerThrottleComponent must be set to 1 on EVERY ESX HOST in the cluster via the following esx cli command.

esxcfg-advcfg -s 1 /VSAN/DOMNetworkSchedulerThrottleComponent

The config option is available in all versions of ESX which support ESA (namely ESX 8.0 onwards). The configuration option can be set on a running host. The effect will be immediate and persistent. Setting the config on hosts with networking faster than 10g will not cause functional issues but may result in suboptimal resynchronization throughput.