All VM Tasks Hang on the vCenter or on the Host

Products

• VMware vSphere ESXi

Issue/Introduction

• All the VM related tasks such as Power ON, Snapshot Consolidation, vMotion in vCenter or in the Hosts gets hung and does not proceed further.
• Under the ESXi Host's /var/run/log/vmkernel.log, below error log snips are available:
  
  
  • "Hotplug" events from the HBA driver:

    YYYY-MM-DDTHH:MM:SS In vmkernel: cpu2:#######)lsi_mr3_0000:10:00.0: megasas_hotplug_work: 580: event  code: 0x71

  • High latency reported:

    YYYY-MM-DDTHH:MM:SS In vmkwarning: cpu22:#######)WARNING: ScsiDeviceIO: 1780: Device naa.################### performance has deteriorated. I/O latency increased from average value of 15728 microseconds to 2859553 microseconds.

  • SCSI command aborts:

    YYYY-MM-DDTHH:MM:SS cpu14:######)ScsiDeviceIO: 4321: Cmd(0x############) 0x2a, CmdSN 0xabcd to dev "t10.NVMe____####################" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

  • Additional hardware errors, such as:
    
    YYYY-MM-DDTHH:MM:SS cpu14:######)WARNING: NVMEMGMT : #### Controller 256 is not in operational state

     

• The VMs are located on a local VMFS datastore.

Cause

This issue occurs when the local disks experience intermittent connectivity with the HBA or fail to respond to SCSI commands.

Resolution

Option 1: Ensure Hard Disk Drive is properly connected/seated in the host.

Option 2: Verify the local HBA/RAID controller has supported drivers and firmware installed are as per the Broadcom Compatibility Guide.

For more information, refer: Broadcom Compatibility Guide & Determining Network/Storage firmware and driver version in ESXi

Option 3: Contact Hardware vendor for further assistance.