Post the patch upgrade the ESXi host, the shared NVMe devices are no longer visible

Products

• VMware vSphere ESXi

Issue/Introduction

This article aims to offer direction and steps to troubleshoot issues concerning the connectivity of NVMe storage between an ESXi host and storage

Upon patch upgrading the ESXi host, the NVMe namespaces and datastores that were previously visible are not appearing in the vSphere Client.

From vmkernel logs we see the below events.

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:177 vmkParams.hostid:  ########-####-####-####-########6410

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:178 vmkParams.hostnqn: nqn.2014-08.in.xxxx.####.####:nvme:####-ESX01

2024-02-09T07:08:58.547Z cpu1:2098192)WARNING: NVMFDEV:882 Failed to transport connect controller 257, status: Failure

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:179 vmkParams.subnqn:  nqn.1988-11.com.dell:powerstore:00:###########

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:190 vmkParams.trType:  FC

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:192 vmkParams.trwwnn:  ############

2024-02-09T07:08:58.547Z cpu1:2098192)NVMEDEV:1565 Ctlr 257 freeing

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:194 vmkParams.trwwpn:  ############

2024-02-09T07:08:58.547Z cpu1:2098192)NVMEDEV:9067 Cancel requests of controller 257, 0 left.

2024-02-09T07:08:58.547Z cpu12:2098193)WARNING: lpfc: lpfc_remoteport_add:1875: 0:6067 No match ############

2024-02-09T07:08:58.547Z cpu12:2098193)WARNING: lpfc: lpfcGetRemotePort:2491: 0:1073 Get Remote Port Failed

2024-02-09T07:08:58.547Z cpu26:2098194)NVMEDEV:1393 Ctlr 259 allocated, maximum queues 16

2024-02-09T07:08:58.547Z cpu12:2098193)WARNING: lpfc: lpfcConnectCtrlr:2543: 0:1076 Failed in Remote Port Get WWNN : ########### WWPN : ###########

2024-02-09T07:08:58.547Z cpu12:2098193)WARNING: NVMFDEV:882 Failed to transport connect controller 258, status: Failure

2024-02-09T07:08:58.547Z cpu1:2098192)WARNING: NVMFEVT:1084 Failed to connect controller nqn.1988-11.com.####:########:00:#############, status: Failure

 

Environment

VMware vSphere ESXi 6.x

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

Post the ESXi host patch upgrade, there was an alteration in the host nqn, which is derived from the fqdn. In this instance, the host fqdn is “ABC-DEF-ESX01.domain.corp.co.in”.

Before the upgrade, the host nqn was “nqn.2014-08.com.vmware:nvme:xxxxxx”. However, following the upgrade, the host nqn transitioned to “nqn.2014-08.in.co.corp.domain:nvme:ABC-DEF-ESX01”. Due to the lack of updates in the initiator section of the storage console reflecting these changes, the ESXi host was unable to establish a session with the storage.

Resolution

Update the initiators in the storage console with the latest host nqn i.e. from  “nqn.2014-08.com.vmware:nvme:xxxxxx” to latest “nqn.2014-08.in.co.corp.domain:nvme:ABC-DEF-ESX0

Additional Information

The NVMe namespaces and datastores can become invisible in the vSphere Client