HPE ESXi server "Lost connectivity to storage" for vmhba33 device

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

• vSphere Client reports a host alert: "Lost connectivity to storage device mpx.vmhba33:C0:T0:L0"
  
  
• The alert often specifies: "Affected datastores: Unknown."
  
  
• The issue frequently occurs shortly after an ESXi host is placed into Maintenance Mode or during upgrade procedures.
  
  
• The device vmhba33 may not be visible in the standard storage adapter view in the vSphere Client.
  
  
• The device is reported to have entered an APD state under tasks and events: "Device or filesystem with identifier mpx.vmhba33:C0:T0:L0 has entered the All Paths Down State."

 

 

Environment

VMware vSphere ESXi 8.x

Cause

• This arises where vmhba33 is a virtual media device (Virtual CD-ROM) presented by the HPE Integrated Lights-Out (iLO) controller.
• This device is often automatically attached or detached by hardware management software, such as HPE OneView, during maintenance workflows or firmware updates.
• When the device is detached by the hardware, ESXi triggers a storage connectivity alarm because the path to the previously registered CD-ROM device is lost.

 

That device vmhba33 is a virtual CD-ROM can be confirmed as follows:

esxcfg-info -a

\==+LUN :
                 |----Name...................................mpx.vmhba33:C0:T0:L0
                 |----External Id............................mpx.vmhba33:C0:T0:L0
                 |----Type...................................CD-ROM
                 |----Vendor.................................iLO
                 |----Model..................................Virtual CD-ROM
                 |----Revision...............................0001
                 |----Display Name...........................Local USB CD-ROM (mpx.vmhba33:C0:T0:L0)
                 |----Has Settable Display Name..............false
                 |----Path Plugin............................NMP

 

esxcli storage core device list

mpx.vmhba33:C0:T0:L0:
   Display Name: Local USB CD-ROM (mpx.vmhba33:C0:T0:L0)
   Has Settable Display Name: false
   Size: 0
   Device Type: CD-ROM
   Multipath Plugin: NMP
   Devfs Path:
   Vendor: iLO
   Model: Virtual CD-ROM
   Revision: 0001
   SCSI Level: 2
   Is Pseudo: false
   Status: dead timeout

 

We can see it gets discovered and registered and by the vmkusb module.

ESXi /var/log/vmkernel.log
####-##-##T##:##:##.###Z In(182) vmkernel: cpu56:2098396)vmkusb: uhub_explore:1236: illegal enable change, port 1
####-##-##T##:##:##.###Z In(182) vmkernel: cpu73:2098396)vmkusb: umass_attach:1132: umass_attach: Attach device cached_name NULL, cached data ff
####-##-##T##:##:##.###Z In(182) vmkernel: cpu73:2098396)vmkusb: uhub_explore:1236: illegal enable change, port 2
####-##-##T##:##:##.###Z In(182) vmkernel: cpu73:2098160)vmkusb: umass_watchdog:1017: umass_watchdog: Register SIM for New Device with 0 sec(s) delay
####-##-##T##:##:##.###Z In(182) vmkernel: cpu55:2098140)ScsiDevice: 4970: Successfully registered device "mpx.vmhba33:C0:T0:L0" from plugin "NMP" of type 5

Resolution

 

To determine if this is the cause or to mitigate the alerts, perform the following:

Connectivity alerts for vmhba33 are often tied to how the physical hardware presents virtual media to the operating system. Investigate the following areas to identify and resolve the cause:

1. HPE OneView/Management Orchestration: Review the Server Profile or Maintenance Policy in HPE OneView. Verify if the workflow includes mounting or unmounting an ISO/Virtual Media when the host enters Maintenance Mode.
2. iLO Virtual Media State:
   • Access the iLO web interface for the host during the maintenance window.
   • Navigate to Virtual Media and monitor if the device status changes from 'Connected' to 'Disconnected' automatically.
     
     
3. Hardware Vendor Engagement: If the device is disconnecting unexpectedly without an associated management workflow, engage the hardware vendor to inspect the iLO firmware and hardware logs.