Storage Latency Due to Bandwidth Saturation on Storage Array Ports
Article ID: 445037
Updated On:
Products
Issue/Introduction
High storage latency metrics on ESXi hosts can often be traced to fabric congestion or insufficient physical pathing during peak I/O workloads.
Symptoms can include:
- ESXi hosts report high storage latency metrics (e.g., 100ms or higher) across multiple Network Addressing Authority (NAA) identifiers.
- Performance degradation is observed during heavy asynchronous workloads or scheduled tasks.
- Network pause frames may be logged, indicating fabric congestion or buffer credit starvation.
- Latency events are concentrated on specific LUNs but appear systemic across the fabric.
Environment
VMware vSphere ESXi 8.x/7.x
Cause
The latency is caused by an interface or path-level oversubscription bottleneck. When the aggregate throughput of the existing physical ports between the storage array and the Fabric Interconnects (or switches) is exceeded, buffer queues fill up. This triggers hardware-level flow control (pause frames), which cascades latency back to the ESXi hosts.
Resolution
To resolve the port saturation and scale aggregate throughput, increase the available bandwidth from the storage array:
- Add Physical Paths: Execute an infrastructure plan to add additional physical ports from the storage array to the Cisco UCS Fabric Interconnects (FI) or upstream switches.
- Expand Link Aggregation: Ensure the new ports are correctly added to the existing link aggregation group capacity.
- Balance I/O Load: Verify that the sub-LUN I/O load is spread across the newly available paths to prevent future front-end port oversubscription.
Monitor the storage response times during high-workload events to confirm that the increased bandwidth has stabilized systemic latency and eliminated the occurrence of pause frames.
Additional Information
Feedback
