High Write Latency in vSAN OSA: Log Buffer Saturation and Network TCP Congestion

Products

• VMware vSAN

Issue/Introduction

• High I/O wait time is observed.
• On the ESXi host, a write latency is recorded on the DOM disk group.
• No latency is observed at the LSOM layer for the same cache disk. However, log congestion bandwidth is detected.
• vCenter Performance Diagnostic alert - "The increase in latency in the vSAN stack might be beyond expected limits"
• vmkernel logs may have "RPC to DOM op readPolicy returned: No connection."

Environment

• VMware vSAN OSA 8.x

Cause

• The primary cause of the congestion is the rapid saturation of the vSAN write log buffer. This occurs because a large volume of small write I/O operations are simultaneously targeting many different vSAN components within the same disk group.
• This pattern leads to two specific issues:
  • The small, scattered writes quickly fill the cache tier's log buffer.
  • When the log buffer is full, vSAN must pause or throttle new incoming writes until space is cleared by destaging data to the capacity tier. This throttling directly causes the measured high write latency.
• Outstanding I/O operations will be recorded in the same disk group at the time of the issue, confirming heavy queue buildup.
• LogP (log buildup) will reach 100%, meaning the log buffer space is fully utilized. This correlates with the burst of I/O observed during the incident window.
• This indicates that an I/O burst occurred and is sustained for a short period.
• TCP retransmissions and TCP Out-Of-Order (OOO) packets can prevent vSAN DOM layer from acknowledging I/O quickly enough, which can then cause the log buffer to saturate faster than normal

Resolution

The goal is to increase the vSAN ability to process and destage small write I/O operations to prevent log buffer saturation.

1. Analyze the virtual disk metrics for all VMs in the affected disk group. Specifically, identify which VMDKs are generating the large volume of frequent, small write I/O operations.

2. Collaborate with the application team to understand the application's I/O behavior. Determine the function of these high-IOPS virtual disks (e.g., database transaction logs, temporary file systems, or message queues).

3. After identifying the virtual disks causing the issue, divide the VMDKs into smaller ones. This helps vSAN spread the objects across more disk groups and reduces the load on any single disk group.

4. Review and apply vSAN storage policies optimized for high IOPS workloads, such as using fewer components per object and adjusting stripe width or FTT settings appropriately.

5. Ensure that the Application and the guest operating system's file system I/O is correctly aligned to 4K block sizes.

     6. Verify NICs and BIOS, driver\firmware are up to date.

     7. To validate any TCP drops and\or TCP Out-Of-Order packets, use tcpdump. See Packet Captures in ESXi using tcpdump 

    

Additional Information

Also See: Understanding vSAN Network TCP/IP Expectations and Thresholds