ESXi Hosts Report Persistent Storage Connectivity Alarms for Defunct or Stale Devices

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms: 

• Alert with error sent via email to administrator as below,

  MN Monitoring Notifications First User Email New To: [email protected] This email is to notify you that an alarm has been triggered in your vCenter: Alarm alarm.StorageConnectivityAlarm on Host ###-###-###-##.##.##### because Lost connectivity to storage device eui.################################. Path vmhba##:C#:T#:L# is down. Affected datastores: Unknown .. Alarm name alarm.StorageConnectivityAlarm Description alarm.StorageConnectivityAlarm Target Host ###-###-###-##.##.##### Triggered time ##/##/#### ##:##:## PM.

• The Alerts being triggered for stale Devices which are already decommissioned.

• Hosts UI displays events with below error, Select Esxi host -> Monitor -> Tasks and Events -> Events

  "Device eui.eui.45e95096ab0232cc6c9ce900f860e9f8 has been removed or is permanently inaccessible. Affected datastore (if any): unknown".

Environment

VMware vSphere ESXi 8.x

Cause

• A non-existent device ID triggering alarms in vCenter is usually caused by stale configuration data (ghost devices), The alarm persists because vCenter is trying to report on a device that was removed or failed but was not cleanly unregistered from the vCenter inventory.

• The logs from /var/run/log/vmkernel.log, reports error inaccessible devices.

####-##-##T##:##:##.078Z Wa(180) vmkwarning: cpu37:3165525)WARNING: ScsiDevice: 1794: Device :eui.################################### has been removed or is permanently inaccessible.

####-##-##T##:##:##.078Z In(182) vmkernel: cpu37:3165525)ScsiDevice: 1808: Permanently inaccessible device :eui.################################### has no more open connections. It is now safe to unmount datastores (if any) and delete the device.

Note: Same error appears for other Stale devices

• The logs of /var/run/log/hostd.log displays that stale devices cached in the ConfigStore are causing host reconnection attempts to fail. The host is incorrectly attempting to initialize paths for these defunct devices.

  ####-##-##T##:##:##.012Z In(166) Hostd[2099550]: [Originator@6876 sub=Libs opID=esxcli-##-da## sid=52c3f82f user=root] info [ConfigStore:##########] Checking for empty objects and arrays in comp esx grp storage key block_devices id eui.################################ object

  ####-##-##T##:##:##.013Z In(166) Hostd[2099550]: [Originator@6876 sub=Libs opID=esxcli-78-da08 sid=52c3f82f user=root] info [ConfigStore:##########] [cs:1:611431074]BeginTransaction invoked.

  ####-##-##T##:##:##.013Z In(166) Hostd[2099550]: [Originator@6876 sub=Libs opID=esxcli-78-da08 sid=52c3f82f user=root] info [ConfigStore:##########] [cs:1:611431074]Transaction started, level = 2

  ####-##-##T##:##:##.013Z In(166) Hostd[2099550]: [Originator@6876 sub=Libs opID=esxcli-78-da08 sid=52c3f82f user=root] info [ConfigStore:##########] Checking for empty objects and arrays in comp esx grp storage key block_devices id eui.################################ object

Resolution

Reboot the ESXi hosts in the cluster to clear stale devices from host configurations, if the issue still persists please open a support case with Broadcom Support.