Reference for vSAN resync wait period and ESXCLI commands after host network or NIC failure

Products

• VMware vCenter Server

Issue/Introduction

After an ESXi host loses vSAN network connectivity, experiences a NIC failure, or is placed into maintenance mode, vSAN objects may show as reduced availability, absent, stale, degraded, or resyncing. The vSAN UI may show a large amount of data left to resync and the value may fluctuate while vSAN calculates, repairs, or evacuates components.

• The vSAN repair delay / object repair timer wait period
• Whether resync throttling affects VM I/O or vSAN resync traffic
• How to check if resync throttling is configured
• Which ESXCLI commands can be used to monitor resync and vSAN host/network health

Environment

vCenter Server 7.x, 8.x, 9.x
vSAN OSA
vSAN ESA

 

Cause

The vSAN repair delay timer applies to host failure or maintenance-mode scenarios. It is not a general delay for all device failures. If a physical cache or capacity disk failure is detected, vSAN can begin rebuilding immediately.

During this process, the amount of data remaining to resync can change as vSAN recalculates object placement, discovers additional stale or absent components, processes active VM write I/O, or continues maintenance-mode evacuation.

The vSAN repair delay timer controls when vSAN begins repair after certain host failure conditions. It does not speed up an active resync that has already started.

Resolution

1. Understand the vSAN repair delay / object repair timer
   • The default vSAN repair delay is 60 minutes. This wait period gives vSAN time to determine whether a host failure is transient or permanent before rebuilding absent components.
     • Do not reduce the repair delay timer as a method to speed up an active resync. Once vSAN repair or resync is already in progress, changing the timer does not make the current resync complete faster.
     • If the repair delay is changed through the vSphere UI, wait at least 180 seconds before starting maintenance, disk group, disk, or reboot-related activity.
     • After the failure or maintenance activity is complete, reset the repair delay timer to the default value unless there is a specific reason to keep a different value.
   • To check the current repair delay value on an ESXi host:
     • esxcli system settings advanced list -o /VSAN/ClomRepairDelay
   • To reset the repair delay value to 60 minutes on an ESXi host:
     • esxcli system settings advanced set -o /VSAN/ClomRepairDelay -i 60
     • *Note: Only change this setting when there is a clear operational reason. Ensure the value is consistent across the vSAN cluster.
2. Understand vSAN resync throttling
   • The vSAN resync throttle controls vSAN resync/rebuild traffic, it does not directly throttle normal VM I/O. The setting caps vSAN resync traffic, not normal VM I/O. Heavy resync traffic can still contribute to congestion and VM latency if the environment is bottlenecked.
     • A throttle value of 0 means no manual resync throttle is configured.
   • To check the current vSAN resync throttle value:
     • esxcli vsan resync throttle get
       • Example output:
         • Level: 0 Mbps
           • This indicates vSAN resync traffic is not manually capped by this setting.
   • To remove a manual resync throttle:
     • esxcli vsan resync throttle set --level=0
   • To configure a manual resync throttle:
     • esxcli vsan resync throttle set --level=<0-512>
       • Example:
         • esxcli vsan resync throttle set --level=200
     • *Note: Use throttling only when resync traffic is causing unacceptable VM latency or network/storage contention. Applying a throttle can reduce the impact on running workloads, but it can also extend the time that objects remain reduced redundancy or non-compliant.
3. Check current resync progress (debug commands and output can vary by release)
   • Run the following command from an ESXi host in the vSAN cluster:
     • esxcli vsan debug resync summary get
   • Use this command to view a more detailed list of resyncing objects/components:
     • esxcli vsan debug resync list
   • Check the current resync bandwidth:
     • esxcli vsan resync bandwidth get
   • Check object health summary:
     • esxcli vsan debug object health summary get
   • Recommended monitoring practice:
     • Take multiple samples 30 to 60 minutes apart. Do not rely on a single sample because the number of objects and GB left to resync can fluctuate while vSAN recalculates and processes the queue.
4. Validate vSAN host and network health
   • Check vSAN network configuration:
     • esxcli vsan network list
   • Check vSAN cluster membership:
     • esxcli vsan cluster get
   • Check vSAN unicast agent entries:
     • esxcli vsan cluster unicastagent list
   • Check physical NIC status:
     • esxcli network nic list
   • Check physical NIC statistics and errors:
     • esxcli network nic stats get -n vmnicX
       • Replace vmnicX with the physical adapter used by the vSAN network.
   • Validate basic vSAN VMkernel connectivity:
     • vmkping -I vmkX <peer-vSAN-vmk-IP>
     • Validate jumbo frame MTU if the vSAN network uses MTU 9000 or 1500:
       • vmkping -I vmkX -d -s 8972 <peer-vSAN-vmk-IP>
       • vmkping -I vmkX -d -s 1472 <peer-vSAN-vmk-IP>
5. Interpret increasing or fluctuating GB left to resync
   • The “GB left to resync” value can fluctuate. An increase does not always mean the resync failed.
     • Possible causes for an increase:
       • vSAN discovered additional objects or components that require resync.
       • A host, disk, disk group, or vSAN network path became unstable again.
       • Active VM write I/O continued while objects were reduced redundancy.
       • vSAN recalculated object placement or maintenance-mode evacuation work.
       • A storage policy or compliance operation added additional resync work.
       • The UI updated after vSAN calculated a larger set of affected objects.
         • If the value increases over a long period, collect multiple samples and correlate the increase with vCenter events, vSAN health, host maintenance-mode activity, pNIC errors, disk health, and VM/storage workload activity.

Additional Information

KB 327006: Heavy resync traffic may cause VM I/O performance degradation
KB 327031: Changing the default repair delay time for a host failure in vSAN
KB 326955: Understanding host level resync management
KB 326963: vSAN Health Service - Resync operations throttling
ESXCLI Command Reference: esxcli vsan commands