vSAN VM Latency and SSD Congestion Following Extended Host Maintenance Window

Products

• VMware

Issue/Introduction

• Virtual objects report the health state: Reduced availability with no rebuild - delay timer.
• Production Virtual Machines (including vCenter Server) experience significant disk latency and performance degradation.
• Multiple PROACTIVE_REBALANCE and PLACEMENT workitems are observed in vSAN management logs.
• vSAN Health logs show objects in a datamove state.

Environment

VMware vSAN 8.x

Cause

• The issue is caused by a large-scale data resynchronization and proactive rebalance operation triggered when hosts rejoin the cluster after an extended period in Maintenance Mode.
• If hosts remain in Maintenance Mode beyond the default vSAN repair delay (typically 60 minutes), vSAN may begin rebuilding components elsewhere to restore compliance. When the original hosts return, vSAN initiates significant data movement to redistribute objects and reconcile accumulated data changes.
• This sudden burst of background I/O can overwhelm the SSD cache tier write buffer, leading to SSD congestion and subsequent latency for production VM workloads.

Resolution

To prevent SSD congestion and production impact during maintenance operations, follow these steps:

1. To protect VM I/O, manually limit the background resync throughput on all hosts in the cluster. Apply this throttle in either of the following scenarios:

   • Before a host exits Maintenance Mode (especially if it has been offline for an extended period).

   • If the cluster is already experiencing high SSD congestion after a host has exited Maintenance Mode.

     
     
     

      

2. Run the following command on each ESXi host in the cluster:
   
   • esxcli vsan resync throttle set --level <0-512mb>
   • Understanding host level resync management
3. To avoid large-scale rebuilds, attempt to complete updates and return hosts to the cluster within the default 60-minute Object Repair Timer window.
   
   
4. Use the vSAN Performance Service to monitor SSD Congestion and Resync Latency via the vSphere Client under Monitor > vSAN > Performance.
   
   
5. Consider SSD Cache Size, vSAN 8.0 supports larger write buffers for cache disks, which significantly improves the system's ability to absorb heavy I/O bursts without triggering congestion.
   VMware vSAN Support for large cache tier write buffer in vSAN 8.0 and higher