Managing NVMeoF adapters created on deprecated HW devices
Article ID: 313236
Updated On:
Products
Issue/Introduction
Symptoms:
Users might see stale NVMeoF connection entries when listed.
A stale connection entry is an entry in the connection list that does not correspond to a valid NVMe adapter. For instance, consider the following:
# esxcli nvme adapter list
Adapter Adapter Qualified Name Transport Type Driver Associated Devices
------- -------------------------------------------------------- -------------- --------- ------------------
vmhba0 aqn:nvme_pcie:nqn.2021-08.com.intel:PHAB134101GA3P8EGN PCIe nvme_pcie
vmhba1 aqn:nvme_pcie:nqn.2021-04.com.intel:PHAB1154005U1P9SGN PCIe nvme_pcie
vmhba68 aqn:nvmerdma:24-8a-07-96-4d-8e RDMA nvmerdma vmrdma4, vmnic7
vmhba69 aqn:nvmerdma:24-8a-07-96-4d-8f RDMA nvmerdma vmrdma5, vmnic8
vmhba71 aqn:nvmetcp:24-8a-07-96-4d-8e-T TCP nvmetcp vmnic7
vmhba72 aqn:nvmetcp:24-8a-07-96-4d-8f-T TCP nvmetcp vmnic8
# esxcli nvme fabrics connection list
Adapter Qualified Name Admin Queue Size Connection Mode Controller ID Digest Is CDC Keep Alive Timeout Subsystem NQN Transport Address Transport Type
------------------------------- ---------------- ---------------- ------------- ------ ------ ------------------ ------------------------------------------------------- ----------------- --------------
aqn:nvmetcp:00-02-c9-b5-e5-30-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.21 TCP
aqn:nvmetcp:24-8a-07-96-4d-8e-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.21 TCP
aqn:nvmerdma:24-8a-07-96-4d-8f SINGLE_SUBSYSTEM false nqn.2010-06.com.purestorage:flasharray.4d4bafbf03558e0f 10.20.54.101 RDMA
aqn:nvmetcp:00-02-c9-b5-e5-30-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.22 TCP
aqn:nvmetcp:24-8a-07-96-4d-8f-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.22 TCP
aqn:nvmetcp:24-8a-07-96-4d-8e-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.22 TCP
aqn:nvmerdma:24-8a-07-96-4d-8e SINGLE_SUBSYSTEM false nqn.2010-06.com.purestorage:flasharray.4d4bafbf03558e0f 10.20.54.101 RDMA
aqn:nvmetcp:24-8a-07-96-4d-8f-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.21 TCP
aqn:nvmerdma:00-02-c9-b5-e5-30 SINGLE_SUBSYSTEM false nqn.2010-06.com.purestorage:flasharray.4d4bafbf03558e0f 10.20.54.101 RDMA
In the above example, all connection entries with Adapter Qualified Name of "aqn:nvmetcp:00-02-c9-b5-e5-30-T" and "aqn:nvmetcp:00-02-c9-b5-e5-30" are stale as they do not correspond to any valid NVMeoF adapter in the system.
Environment
Cause
As part of upgrade preparation, users should identify any planned deprecated devices for the next ESXi version (as mentioned in KB 88172) and disable the respective NVMeoF adapters that use these deprecated devices prior to the ESXi update.
Follow the steps to disable the NVMeoF adapter.
# esxcli nvme fabrics disable -p <protocol> -d <device>
For example
# esxcli nvme fabrics disable -p RDMA -d vmrdma2
# esxcli nvme fabrics disable -p TCP -d vmnic6
In case user has not performed this step prior to update, then after upgrade user might observe stale connection entries.
Resolution
Workaround:
To workaround the issue, clean up the stale entries following the below mentioned steps:
Removing connection entry
# esxcli nvme fabrics connection delete -a <Adapter Qualified Name>"
In the above example, the stale entries can be cleaned up by running
# esxcli nvme fabrics connection delete -a aqn:nvmetcp:00-02-c9-b5-e5-30-T
# esxcli nvme fabrics connection delete -a aqn:nvmetcp:00-02-c9-b5-e5-30
To confirm the entries are gone from the system,
# esxcli nvme fabrics connection list
Adapter Qualified Name Admin Queue Size Connection Mode Controller ID Digest Is CDC Keep Alive Timeout Subsystem NQN Transport Address Transport Type
------------------------------- ---------------- ---------------- ------------- ------ ------ ------------------ ------------------------------------------------------- ----------------- --------------
aqn:nvmetcp:24-8a-07-96-4d-8e-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.21 TCP
aqn:nvmerdma:24-8a-07-96-4d-8f SINGLE_SUBSYSTEM false nqn.2010-06.com.purestorage:flasharray.4d4bafbf03558e0f 10.20.54.101 RDMA
aqn:nvmetcp:24-8a-07-96-4d-8f-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.22 TCP
aqn:nvmetcp:24-8a-07-96-4d-8e-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.22 TCP
aqn:nvmerdma:24-8a-07-96-4d-8e SINGLE_SUBSYSTEM false nqn.2010-06.com.purestorage:flasharray.4d4bafbf03558e0f 10.20.54.101 RDMA
aqn:nvmetcp:24-8a-07-96-4d-8f-T SINGLE_SUBSYSTEM false nqn.1988-11.com.dell:powerstore:00:18c4eb9a7a2bC0891214 10.20.2.21 TCP
Feedback
