ESX host alarm reports datastore connectivity failure as "Lost access to volume UUID due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly"
search cancel

ESX host alarm reports datastore connectivity failure as "Lost access to volume UUID due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly"

book

Article ID: 450696

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Alarm notifies users about datastore connectivity status. The alarm reads, "Lost access to volume UUID (datastore1) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly".

As the alert message require more clarity, considering reviewing the status of the datastore in connected state with active paths. 

The investigation begins with analyzing vmkernel logs and esxtop values, wherein, the commands such as 0x2a and 0x8a shows in transit failures or command reset. The esxtop shows high DAVG values.

/var/log/vmware/vmkernel.log shows write commands (0x2a) and (0x8a) being cancelled

YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu37:2097258)ScsiDeviceIO: 4524: Cmd(0x45b90ace76c0) 0x2a, cmdId.initiator=#### CmdSN 0x325ba8 from world #### to dev "naa.id" failed H:0x8 D:0x0 P:0x0 Cancelled from device layer
YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu32:2098207)lsi_mr3_0000:c3:00.0: mfi_VirtReset: 645: VIRT_REST completed. Initiator ID #### World ID ####
YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu32:2098207)lsi_mr3_0000:c3:00.0: mfi_TaskMgmt: 689: Processing taskMgmt virt reset for device: vmhba2:C3:T111:L0
YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu32:2098207)lsi_mr3_0000:c3:00.0: mfi_TaskMgmt: 696: Virtual Reset request from Wld 2102162
YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu37:2097258)ScsiDeviceIO: 4524: Cmd(0x45b90496bc00) 0x8a, cmdId.initiator=#### CmdSN 0x800e0038 from world 2102162 to dev "naa.id" failed H:0x8 D:0x0 P:0x0 Cancelled from device
YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu32:2098207)lsi_mr3_0000:c3:00.0: mfi_VirtReset: 630: VR : SMID 65 IN 2084516s OUT 0s TM 2084527s Delta 11s
YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu37:2097258)ScsiDeviceIO: 4524: Cmd(0x45b90796b740) 0x8a, cmdId.initiator=#### CmdSN 0x800e0005 from world 2102162 to dev "naa.id" failed H:0x8 D:0x0 P:0x0 Cancelled from device
YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu37:2097258)ScsiDeviceIO: 4524: Cmd(0x45b8ff13eb40) 0x8a, cmdId.initiator=#### CmdSN 0x800e0010 from world 2102162 to dev "naa.id" failed H:0x8 D:0x0 P:0x0 Cancelled from device
YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu37:2097258)ScsiDeviceIO: 4524: Cmd(0x45b90ad3ba40) 0x8a, cmdId.initiator=#### CmdSN 0x800e001d from world 2102162 to dev "naa.id" failed H:0x8 D:0x0 P:0x0 Cancelled from device
YYYY-MM-DDTHH:MIN:SEC.## In(182) vmkernel: cpu37:2097258)ScsiDeviceIO: 4524: Cmd(0x45b8ff057040) 0x8a, cmdId.initiator=#### CmdSN 0x800e0019 from world 2102162 to dev "naa.id" failed H:0x8 D:0x0 P:0x0 Cancelled from device

Cause

High workload towards storage controller can lead to such behavior.

Resolution

Review storage device status from Hardware console and check if a device drive is faulty. This might lead to high IOPS towards storage controller managing the devices. The device driver and firmware, thus plays a vital role in load balancing. Validate if Driver/firmware is compatible as Broadcom HCL. If differences found, update the same.

Additional Information

Further reading:

Powered by Wolken Software