ESX hosts attached to IBM Storage Virtualize NVMe storage may log errors related to opcode 0x5

Products

• VMware vSphere ESXi

Issue/Introduction

/var/log/vmkernel.log is being flooded for periods of time with messages relating to compare commands, NVMe OpCode = 0x5.


Please see a representative /var/log/vmkernel.log snippet:

Wa(180) vmkwarning: cpu35:5393358)WARNING: NVMEIO:2645 command 0x45baa7b56040 failed: ctlr 264, queue 7, psaCmd 0x45bab2fb3d40, status 0x9, opc 0x5, cid 28, nsid 7
Wa(180) vmkwarning: cpu35:5393358)WARNING: NVMEIO:2645 command 0x45baa7b76e40 failed: ctlr 264, queue 7, psaCmd 0x45bab2fed940, status 0x9, opc 0x1, cid 5, nsid 7
Wa(180) vmkwarning: cpu35:5393358)WARNING: lpfc : vmhba3 nvmeFcCmdCompletion:845: 1028 Command NVMe Status : 0x9 Opcode : 0x5
Wa(180) vmkwarning: cpu35:5393358)WARNING: NVMEIO:2645 command 0x45baa7b64c40 failed: ctlr 264, queue 8, psaCmd 0x45bab2fb3d40, status 0x9, opc 0x5, cid 44, nsid 7
Wa(180) vmkwarning: cpu35:5393358)WARNING: lpfc : vmhba3 nvmeFcCmdCompletion:845: 1028 Command NVMe Status : 0x9 Opcode : 0x5
Wa(180) vmkwarning: cpu35:5393358)WARNING: lpfc : vmhba3 nvmeFcCmdCompletion:845: 1028 Command NVMe Status : 0x9 Opcode : 0x5
Wa(180) vmkwarning: cpu35:5393358)WARNING: lpfc : vmhba3 nvmeFcCmdCompletion:845: 1028 Command NVMe Status : 0x9 Opcode : 0x5

 

The compare command (opcode 0x5) may be part of a fused command e.g. a compare (0x5) and write (0x1). In thes failure of the compare command will lead to failure of the write command also, and of the fused command as a whole.

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x

Cause

Known vendor issue with NVMe devices on IBM Storage Virtualize prior to version 8.7.0.8

Resolution

Please engage with IBM support.

Additional Information

Please see the IBM document: 

Release Note for systems built with IBM Storage Virtualize

See SVAPAR-139491.


For more information on NVMe OpCodes and Status definitions please see the following KB article:

NVMe OpCodes and Status Definitions