Storage Latency during user logon storm

Products

• VMware vSphere ESXi

Issue/Introduction

• Storage latency observed at the VM and LUN level during user login storm every morning for 1 hour.
  
  
•  /var/log/vmkernel.log reports logging similar to:

Device naa.################# performance has deteriorated. I/O latency increased from average value of 249 microseconds to 33256 microseconds

Environment

VMware vSphere ESXi 8.0

Cause

This can arise if one large LUN ( or small number of large LUNs) hosts all production VM's that employees are using for login in the morning. 

This leads to high load and latency on the LUN.

Resolution

Replace one large LUN with multiple smaller LUNs

 

 

Additional Information

Making LUN Decisions

• Architectural Resilience: Deploying multiple FC LUNs prevents single points of failure (corruption, physical disk failure, cyber attacks) and improves risk isolation.
• Queue Parallelism: Multiplying device queues across multiple LUNs eliminates I/O throttling, reduces latency, and maximizes throughput under heavy loads.
• Pathing & Metadata Isolation: Spreading workloads optimizes load balancing across physical HBAs while containing performance impact from metadata-heavy tasks.
• HBA Limit Management: Care must be taken to ensure total aggregate LUN queues do not exceed physical HBA target queue limits and cause bus errors.