Performance latency due to degraded physical disk for vSAN

Products

• VMware vSAN

Issue/Introduction

This article addresses performance degradation and latency within a vSAN cluster. This behavior is typically caused by failing or flapping physical disks that trigger storage path drops and I/O queue bottlenecks.

• High read/write latency reported at the virtual machine level.
• "Unhealthy" status for Physical Disk Operation Health in the vSAN Health dashboard.
• vSAN Object Health alerts indicating degraded objects.
• Massive I/O queues forming on specific hosts.

Environment

• VMware vSAN (All versions)
• Physical Server Hardware (e.g., HPE, Dell, etc.)

Cause

Degraded physical disks specifically those experiencing frequent I/O errors or flapping can cause storage paths to drop (All Paths Down / APD). As the vSAN storage layer repeatedly retries failed I/O requests, it creates an I/O queue bottleneck, resulting in performance degradation across the cluster. This is often noticed during high I/O maintenance tasks like datastore decommissioning.

Resolution

To resolve vSAN latency caused by a degraded disk, follow these steps:

1. Verify vSAN Health: Check vSAN Health monitoring for "Physical disk operation health" and "vSAN object health" alerts.
2. Inspect Hardware Logs: Consult physical host management tools (e.g., iLO, iDRAC) or host logs for disk failure events or predictive failures.
3. Identify Faulty Disk: Cross-reference vSAN health alerts with the hardware console to pinpoint the specific disk ID (e.g., naa.####).
4. Replace Hardware: Once the physical disk is identified as failed or degraded, engage the hardware vendor to replace the drive.
5. Monitor Rebuild: After replacement, the vSAN layer will automatically clear I/O queues and begin rebuilding storage objects. Monitor the resynchronization status until complete.

Additional Information

• For the latest updates and patches, see Download Broadcom products and software.
• To speak with a customer representative see Contact Support. Scroll to the bottom of the page and click on your respective region.