Understanding lost access to volume messages in ESXi
search cancel

Understanding lost access to volume messages in ESXi

book

Article ID: 318897

calendar_today

Updated On:

Products

VMware vSphere ESX 8.x VMware Tanzu Kubernetes Grid

Issue/Introduction

 

Symptoms

  • Virtual machines appear inaccessible or "grayed out" in vCenter Server.
  • Tanzu/TKG specific: Commands like kubectl get pods fail with Error from server: etcdserver: request timed out.
  • In vCenter Server, reference events similar to: Lost access to volume 54f89e21-########-####-##########98 (Datastore_Name) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
  • In/var/log/hostd.log: info hostd[ID] [sub=Vimsvc.ha-eventmgr] Event 15734 : Lost access to volume 5d9b401e-########-####-##########42 (Datastore_Name) due to connectivity issues.
  • In/var/log/vobd.log: [vmfsCorrelator] [esx.problem.vmfs.heartbeat.timedout] 66fdce5d-########-####-##########44 VMFS_Volume_Name
  • In/var/log/vmkernel.log: ScsiDeviceIO: 4686: Cmd(0x45...) 0x2a ... failed H:0x7 D:0x0 P:0x0 WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.600..." state in doubt .
    • Guest OS logs (Linux): task blocked for more than 120 seconds or kernel: watchdog: BUG: soft lockup.
    • File systems remount as read-only.
    • vmkwarning.log contains: Device performance has deteriorated. I/O latency increased... to 2076000 microseconds.

 

 

Environment

  • VMware vSphere ESXi 7.x / 8.x
  • VMware Tanzu Kubernetes Grid (TKG) 2.x

Cause

ESXi hosts monitor VMFS datastores via heartbeats issued every 3 seconds. If an I/O operation does not complete within 16 seconds, the datastore is marked offline.

  • I/O Blocking: When a volume enters the "Lost Access" state, the host blocks all I/O until the heartbeat operation completes.
  • Heartbeat Reclaim: ESXi attempts to reclaim the heartbeat approximately once every second.
  • Impact to VMs: VMFS fails all I/O from virtual machines on the impacted datastore with a DEVICE BUSY status until the heartbeat is reclaimed. Guest operating systems remain online only if they can sustain these high-latency periods.
  • TKG Context: In Tanzu environments, this blocking state often triggers etcd timeouts, as the underlying storage becomes unresponsive to the Kubernetes control plane.

Resolution

  1. Identify the Scope: Determine if the "Lost Access" messages are occurring on datastores hosting critical workloads or on secondary volumes intended for removal.
  2. Log Analysis: Review vmkernel.log for SCSI status H:0x7 (Host error), indicating the host cannot communicate with the storage device.
  3. Connectivity Verification: Verify the health of the storage array and FC/iSCSI switches. Check for path failures or "NMP device state in doubt" warnings.
  4. Volume Decommissioning: If the errors correlate to a volume intended for removal:
    • Evacuate all remaining Virtual Machines.
    • Properly unmount and detach the datastore from all hosts in the cluster to stop heartbeat monitoring and prevent hostd timeouts.
    • Tanzu/TKG Recovery: Resolve the underlying storage latency on the ESXi host first. Once the datastore access is stabilized or the problematic volume is removed, the etcd services will recover automatically.
  5. Fabric Verification: Inspect Fibre Channel switch logs for signal issues, CRC errors, or link flaps on ports connected to host HBAs.
  6. Array Analysis: Coordinate with the storage vendor to analyze metrics for CPU saturation, queue depth, or background controller processes.
  7. HCL Alignment: Verify that HBA drivers and firmware strictly align with the ESXi version per the VMware Compatibility Guide.
  8. Guest OS Thresholds: Ensure Guest OS disk timeout parameters are configured to survive transient storage path failovers or latency spikes.

Additional Information

Powered by Wolken Software