High resync I/O latency does not mean slowness in resync progress

Products

• VMware vSAN

Issue/Introduction

• The purpose is to share the new behavior after the introduction of adaptive resync.
• Resync jobs can issue parallel I/Os according to the number of jobs on each host.
• This parallelism could be distributed among several jobs, or concentrated onto one job.
• When the parallel I/Os are issued only for a few jobs, each job's target disk group could show higher than normal resync I/O latencies.
• This does not mean resyncs are slow.
• The latency comes from queueing delay meanwhile the capacity disk layer is servicing the I/Os as fast as possible since the congestion is high enough to begin to throttle the intensive resync I/Os.
• Before VSAN 6.7, a high resync I/O latency is an indication of a slow resync. After VSAN 6.7, a high resync I/O latency does not translate to a slow resync.

Symptoms:
In performance service/grafana pages/dashboards, the resync's write or read I/O latencies might look higher than normal guest I/Os, e.g. higher than 50ms, sometimes even higher than 100ms.

Environment

VMware vSAN 7.0.x

Cause

The higher latency partially comes from the need to activate delta components created for unplanned failures. Thus, the total number of parallel slots on each host is increased to let these resyncs for delta components finish faster, causing a few periods of high resync I/O latencies, especially if the slots are distributed among the last few remaining resync jobs.

Resolution

This is expected behavior with the introduction of Adaptive Resync.