Commands Aborted alarms and NMP retry loops caused by SCSI Sense Code 0x5 0x25 0x0 on orphaned RDMs / Ghost LUNs
search cancel

Commands Aborted alarms and NMP retry loops caused by SCSI Sense Code 0x5 0x25 0x0 on orphaned RDMs / Ghost LUNs

book

Article ID: 445533

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

    • Your ESXi host triggers intermittent or continuous "Commands Aborted" alarms in vCenter.

    • Your Virtual Machines may experience brief stuns, high latency, or storage unresponsiveness.

    • You will notice frequent structural storage state changes in your /var/run/log/vobd.log file, such as path redundancy degradation or Permanent Device Loss (PDL):


  • [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

    You will also find your /var/run/log/vmkernel.log file flooded with Native Multipathing (NMP) failover attempts, explicitly logging SCSI sense data 0x5 0x25 0x0 (ILLEGAL REQUEST: LOGICAL UNIT NOT SUPPORTED):

    WARNING: NMP: nmpCompleteRetryForPath:356: Retry cmd 0x1a (0x45ba4ae1b680) to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed on path "vmhbaX:CX:TX:LX" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.
    WARNING: NMP: nmpCompleteRetryForPath:391: Logical device "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx": awaiting fast path state update before retrying failed command again...

Environment

VMware vSphere ESXi  8.x

Cause

You will encounter this issue when a physical LUN—frequently utilized as a Raw Device Mapping (RDM) attached to your Virtual Machine—is deleted, unmapped, or masked on your backend storage array before you cleanly detach it from your ESXi host and Virtual Machine.

Because your ESXi host was never instructed to stop routing I/O to the device, it continues to send commands down your existing storage paths. The storage array actively rejects these commands with an ILLEGAL REQUEST (0x5 0x25 0x0), indicating it no longer recognizes the LUN. This forces your ESXi host's multipathing daemon into a frantic, endless retry loop across all alternate paths, which exhausts your host's storage queue and triggers your "Commands Aborted" alarms.

Resolution

To resolve this issue, you must completely unmap the orphaned devices from your Virtual Machine(s) and administratively detach them from your ESXi host to flush the dead paths from your routing table.

Phase 1: Remove the Orphaned RDM from Your Virtual Machine

Note: ESXi will prevent you from detaching a storage device if your VM is still actively configured to use it.

  1. Locate your affected Virtual Machine(s) in the vSphere Client.

  2. Right-click the VM and select Edit Settings.

  3. Locate the Hard Disk that maps to the affected naa. ID.

  4. Click the X icon to remove the Hard Disk.

  5. Check the box to Delete files from datastore. (Note: For RDMs, this safely deletes the .vmdk pointer file on your VMFS volume; it does not delete your physical data).

  6. Click OK.

Phase 2: Detach the Dead Device from Your ESXi Host

  1. Open an SSH session to your affected ESXi host and execute the following esxcli commands:
  2. Mark the device as administratively detached:

     
    esxcli storage core device set --state=off -d naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    

     

  3. Verify the device status is 'off':
  4.  
    esxcli storage core device list -d naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | grep "Status:"
    
  5. Permanently remove the detached device from your host inventory:

  6.  
    esxcli storage core device detached remove -d naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    
  7. Rescan the storage adapters :


     
    esxcli storage core adapter rescan --all
    

Once your rescan completes, your host will cease all I/O retries to the unmapped device, and your storage queue will stabilize.

Additional Information