Post the patch upgrade the ESXi host, the shared NVMe devices are no longer visible

Products

• VMware vSphere ESXi

Issue/Introduction

This article aims to offer direction and steps to troubleshoot issues concerning the connectivity of NVMe storage between an ESXi host and storage

Upon patch upgrading the ESXi host, the NVMe namespaces and datastores that were previously visible are not appearing in the vSphere Client.

From vmkernel.log we see the below events.

[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:177 vmkParams.hostid:  ########-####-####-####-########6410
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:178 vmkParams.hostnqn: nqn.2014-08.in.xxxx.####.####:nvme:####-ESX01
[YYYY-MM-DDTHH:MM:SS] cpu1:2098192)WARNING: NVMFDEV:882 Failed to transport connect controller 257, status: Failure
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:179 vmkParams.subnqn:  nqn.1988-11.com.dell:powerstore:00:###########
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:190 vmkParams.trType:  FC
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:192 vmkParams.trwwnn:  ############
[YYYY-MM-DDTHH:MM:SS] cpu1:2098192)NVMEDEV:1565 Ctlr 257 freeing
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:194 vmkParams.trwwpn:  ############
[YYYY-MM-DDTHH:MM:SS] cpu1:2098192)NVMEDEV:9067 Cancel requests of controller 257, 0 left.
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)WARNING: lpfc: lpfc_remoteport_add:1875: 0:6067 No match ############
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)WARNING: lpfc: lpfcGetRemotePort:2491: 0:1073 Get Remote Port Failed
[YYYY-MM-DDTHH:MM:SS] cpu26:2098194)NVMEDEV:1393 Ctlr 259 allocated, maximum queues 16
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)WARNING: lpfc: lpfcConnectCtrlr:2543: 0:1076 Failed in Remote Port Get WWNN : ########### WWPN : ###########
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)WARNING: NVMFDEV:882 Failed to transport connect controller 258, status: Failure
[YYYY-MM-DDTHH:MM:SS] cpu1:2098192)WARNING: NVMFEVT:1084 Failed to connect controller nqn.1988-11.com.####:########:00:#############, status: Failure

Environment

VMware vSphere ESXi 6.x

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

Post the ESXi host patch upgrade, there was an alteration in the host nqn, which is derived from the fqdn. In this instance, the host fqdn is “ABC-DEF-ESX01.domain.corp.co.in”.

Before the upgrade, the host nqn was “nqn.2014-08.com.vmware:nvme:xxxxxx”. However, following the upgrade, the host nqn transitioned to “nqn.2014-08.in.co.corp.domain:nvme:ABC-DEF-ESX01”.

Due to the lack of updates in the initiator section of the storage console reflecting these changes, the ESXi host was unable to establish a session with the storage.

Resolution

Update the initiators in the storage console with the latest host nqn i.e. from  “nqn.2014-08.com.vmware:nvme:xxxxxx” to latest “nqn.2014-08.in.co.corp.domain:nvme:ABC-DEF-ESX0

Additional Information

The NVMe namespaces and datastores can become invisible in the vSphere Client