High read latency occurs for storage with "cold" tier storage enabled

Products

• VMware vSphere ESXi

Issue/Introduction

Some storage vendors have ability to enable tiered storage. However, using "cold" storage can have significantly higher physical latency than performance-optimized storage. When configured or used incorrectly this can cause latency for virtual machines. 

Environment

VMware ESXi with virtual machines using certain tiered storages. 

Cause

There are multiple factors that can cause storage latency when using tiered storage. 

• Physical Limitations: Cold storage often relies on slower media, such as high-capacity mechanical hard drives (HDDs), which have significantly higher physical latency than performance-optimized storage (e.g., SSDs or flash arrays).
• Access Patterns: These tiers are designed for sequential, high-throughput access during bulk retrievals (rehydration), not the random, transactional I/O typical of a running VM.
• Throughput vs. Latency Optimization: Default configurations sometimes prioritize throughput, which can increase latency for latency-sensitive applications unless specific settings are adjusted, and even those adjustments can't overcome slow physical hardware. 

Resolution

If you are experiencing latency with cold storage, work with your storage vendor to make sure it is optimized for your environment, and the types of applications running on the virtual machines. Also, consider the following:

• Tiering Strategy: Ensure only truly "cold" data (infrequently accessed) resides on the cold tier. Keep active data (indexes, manifests, restore metadata) on a hot or warm tier for fast access.
• Monitor I/O Patterns: Use tools like esxtop to identify which VMs are generating high I/O operations per second (IOPS) and consuming the most storage resources.
• Hardware Validation: Verify that your storage hardware is on the Broadcom Hardware Compatibility List and that drivers and firmware are up to date.
• Storage I/O Control (SIOC): Using SIOC allows administrators to prioritize I/O for certain VMs if latency thresholds are exceeded, but this only manages contention, it doesn't make slow storage fast.