Unable to create Datastore on NVMe over RDMA

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

Unable to create Datastore on NVMe over RDMA.

 

The following error is seen in vCenter:

Cannot change the host configuration. Failed to update disk partitions for /vmfs/devices/disks/eui.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 

Cause

[Hostd.log]
xxxx-xx-xxTxx:xx:xx.xxxZ info hostd[2099870] [Originator@6876 sub=Vimsvc.TaskManager opID=lx4pm3le-30593-auto-nm1-h5:70004442-f1-88-a24e user=vpxuser:xxxx\xxxxx] Task Created : haTask-ha-host-vim.host.DatastoreSystem.createVmfsDatastore-111

[vmkernel.log]
xxxx-xx-xxTxx:xx:xx.xxxZ cpu61:2097987)nvmerdma:979 [ctlr 258, queue 8] event 10
xxxx-xx-xxTxx:xx:xx.xxxZ cpu61:2097987)nvmerdma:2846 [ctlr 258, queue 8] disconnected due to CM event 10
xxxx-xx-xxTxx:xx:xx.xxxZ cpu19:2107550)nvmerdma:693 [ctlr 258, queue 8] cleanup vmkCmd 0x45b9c4bf67c0[32] status 0x371
xxxx-xx-xxTxx:xx:xx.xxxZ cpu19:2107550)WARNING: NVMEIO:2624 command 0x45b9c4bf67c0 failed: ctlr 258, queue 8, psaCmd 0x45b9c1481c48, status 0x371, opc 0x1, cid 32, nsid 111
xxxx-xx-xxTxx:xx:xx.xxxZ cpu19:2107550)WARNING: NVMEPSA:203 Complete vmkNvmeCmd: 0x45b9c4bf67c0, vmkPsaCmd: 0x45b9c1481c48, cmdId.initiator=0xZZZZZZZZZZZZZ, CmdSN: 0x3, status: 0x371
xxxx-xx-xxTxx:xx:xx.xxxZ cpu7:2098212)WARNING: NvmeScsi: 172: SCSI opcode 0x2a (0x45b9c1481c48) on path vmhba64:C0:T0:L100 to namespace eui.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx failed with NVMe error status: 0x371 translating to SCSI error
xxxx-xx-xxTxx:xx:xx.xxxZ cpu7:2098212)WARNING: H:0x5 D:0x0 P:0x0
xxxx-xx-xxTxx:xx:xx.xxxZ cpu7:2098212)WARNING: HPP: HppThrottleLogForDevice:1136: Cmd 0x2a (0x45b9c1481c48, 2100000) to dev "eui.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" on path "vmhba64:C0:T0:L100" Failed:
xxxx-xx-xxTxx:xx:xx.xxxZ cpu7:2098212)WARNING: HPP: HppThrottleLogForDevice:1144: Error status H:0x5 D:0x0 P:0x0 . hppAction = 3
xxxx-xx-xxTxx:xx:xx.xxxZ cpu7:2098212)ScsiDeviceIO: 4124: Cmd(0x45b9c1481c48) 0x2a, CmdSN 0x3 from world 2100000 to dev "eui.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x5 D:0x0 P:0x0
xxxx-xx-xxTxx:xx:xx.xxxZ cpu40:2097604)NVMEPSA:1314 taskMgmt:abort cmdId.initiator=0xZZZZZZZZZZZZZ CmdSN 0x3 world:2100000 controller 258 state:5 nsid:111
xxxx-xx-xxTxx:xx:xx.xxxZ cpu40:2097604)NVMEIO:3904 Ctlr 258, ns 111, tmReq 0x43128e873b60, type 1, initiator 0xZZZZZZZZZZZZZ, sn 0x3, world id 2100000.
xxxx-xx-xxTxx:xx:xx.xxxZ cpu35:2097895)NVMEIO:4488 ctlr 258, queue 1, cid 32, cap 0x1, count 0, found cmd 0x45b9c4bf6600 (initiator 0xZZZZZZZZZZZZZ, serialNumber 0x3, worldID 2100000)
xxxx-xx-xxTxx:xx:xx.xxxZ cpu35:2097895)NVMEIO:4584 Issuing command to cancel cmd 0x45b9c4bf6600 (tag 0x43114a80fea0) on queue 1, tracker 0x43128e813620, cid 32
xxxx-xx-xxTxx:xx:xx.xxxZ cpu35:2097895)NVMEIO:4591 cmd2Abort 0x45b9c4bf6600, opcode 0x1, nsid 111, lba 0, lbc 18

Resolution

In the vmkernel logs, the RDMA connection manager has reported an event 10 or disconnect.

Please engage your fabric and storage vendor to investigate the cause of the disconnection.