ESXi Hosts Loss of Access to Multiple Datastores Due to Storage Disconnection - vSphere ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

In a VMware vSphere environment, all ESXi hosts within a specific cluster unexpectedly lose connection to their shared storage array.

The following behaviors are observed:

• In the vCenter Server UI, alerts and events trigger stating: "Lost access to multiple datastores".

• Virtual machines (VMs) running on the affected datastores may stop responding, freeze, or experience severe I/O timeouts.

• Storage performance graphs drop drastically or fail to populate due to total connectivity breakdown.

Similar logs in /var/run/log/vmkernel.log
YYYY-MM-DDTHH:MM:SS.419Z cpu78:2098730)NMP: nmp_ThrottleLogForDevice:3874: Cmd 0x2a (0#############) to dev "naa.###############################" on path "vmhbaXX:CX:TX:LX" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

Environment

VMware vSphere ESXi 8.x

Cause

This issue is typically triggered by a severe transport path failure or a hardware disruption between the ESXi hosts and the storage array. The root causes generally classify into the following areas:

1. Host Bus Adapter (HBA) Issues: * Outdated or incompatible HBA card firmware/driver combinations.

   • Internal hardware failure or degradation of the HBA initiator card itself .

2. Fabric Transport Failures:

   • Interruptions in the Storage Area Network (SAN) switch zoning, faulty physical transceivers (SFPs), or damaged fiber optic cabling.

   • Transient storage array port resets or controllers failing over ungracefully.

When a Storage Initiator Error occurs or a path drops under severe load conditions, ESXi stops receiving response packets, causing a SCSI command failure that manifests as a cluster-wide datastore loss.

Resolution

Because the failure is occurring uniformly across all hosts in the cluster, the investigation must focus on common fabric infrastructure and the physical/firmware layer:

Engage Storage and Fabric Vendors:

• • Contact your Storage Array Vendor to pull array-level logs. Check for storage controller panics, high congestion, port flapping, or unsolicited failover events.

  • Contact your SAN Switch Vendor to review switch fabric status, look for high CRC error counters on ports, or check for faulty physical paths (SFPs/Cables) connecting the ESXi cluster to the SAN.

Checklist for Multipathing Configuration Use this comparison when determining the appropriate policy for your storage environment:

• FIXED Policy (VMW_PSP_FIXED):
  • Behavior: I/O is sent to one designated "preferred" path.
  • Pros/Cons: No inherent load balancing; performance is limited by the bandwidth of one path.
  • Vendor Context: Rarely recommended for modern SAN arrays. Use only if explicitly required by the storage vendor documentation or for specific legacy compatibility requirements.
• Round Robin (RR) Policy (VMW_PSP_RR):
  • Behavior: I/O is distributed across all active, optimized paths.
  • Pros/Cons: Provides load balancing and higher aggregate bandwidth; enables seamless failover.
  • Vendor Context: The standard recommendation for most modern enterprise arrays (e.g., Dell EMC PowerFlex/Unity, Pure Storage, NetApp).
• Configuration Decision Steps:
  • Consult Vendor Guides: Always reference the specific storage vendor’s VMware Compatibility Guide (VCG) or Best Practices configuration guide.
  • Verify Array Capability: Ensure the array supports ALUA (Asymmetric Logical Unit Access) or Active/Active modes before adopting RR.
  • Validate: Apply configuration changes during scheduled maintenance windows to monitor for potential path thrashing or I/O latency shifts.

Additional Information

• Switching to VMW_PSP_RR is generally considered the best practice to prevent I/O congestion and ensure high availability.
• Always ensure strict parity between hardware firmware and VMware-certified driver versions to maintain system stability.