Post the patch upgrade from ESXi 7.0 P05 to ESXi 7.0 P08 and Later, the shared NVMe devices are no longer visible

Products

• VMware vSphere ESXi

Issue/Introduction

This article aims to offer direction and steps to troubleshoot issues concerning the connectivity of NVMe storage between an ESXi host and storage.

Symptoms:

Upon transitioning from the ESXi 7.0 P05 version to the ESXi 7.0 P08 version, the NVMe namespaces and datastores that were previously visible are not appearing in the vSphere Client.
 

From vmkernel logs we see the below events.

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:177 vmkParams.hostid:  63c546b0-a3be-f04e-1979-00620b346410

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:178 vmkParams.hostnqn: nqn.2014-08.in.hcl.corp.hclt:nvme:DEU-HnD-ESX01

2024-02-09T07:08:58.547Z cpu1:2098192)WARNING: NVMFDEV:882 Failed to transport connect controller 257, status: Failure

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:179 vmkParams.subnqn:  nqn.1988-11.com.dell:powerstore:00:8e92b50b2b14346A6EBD

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:190 vmkParams.trType:  FC

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:192 vmkParams.trwwnn:  58ccf090cfe00853

2024-02-09T07:08:58.547Z cpu1:2098192)NVMEDEV:1565 Ctlr 257 freeing

2024-02-09T07:08:58.547Z cpu12:2098193)NVMFDEV:194 vmkParams.trwwpn:  58ccf0984fea0853

2024-02-09T07:08:58.547Z cpu1:2098192)NVMEDEV:9067 Cancel requests of controller 257, 0 left.

2024-02-09T07:08:58.547Z cpu12:2098193)WARNING: lpfc: lpfc_remoteport_add:1875: 0:6067 No match 58ccf0984fea0853

2024-02-09T07:08:58.547Z cpu12:2098193)WARNING: lpfc: lpfcGetRemotePort:2491: 0:1073 Get Remote Port Failed

2024-02-09T07:08:58.547Z cpu26:2098194)NVMEDEV:1393 Ctlr 259 allocated, maximum queues 16

2024-02-09T07:08:58.547Z cpu12:2098193)WARNING: lpfc: lpfcConnectCtrlr:2543: 0:1076 Failed in Remote Port Get WWNN : x58ccf090cfe00853 WWPN : x58ccf0984fea0853

2024-02-09T07:08:58.547Z cpu12:2098193)WARNING: NVMFDEV:882 Failed to transport connect controller 258, status: Failure

2024-02-09T07:08:58.547Z cpu1:2098192)WARNING: NVMFEVT:1084 Failed to connect controller nqn.1988-11.com.dell:powerstore:00:8e92b50b2b14346A6EBD, status: Failure

 

Environment

VMware vSphere ESXi 7.0

Cause

Post the ESXi host patch upgrade, there was an alteration in the host nqn, which is derived from the fqdn. In this instance, the host fqdn is “ABC-DEF-ESX01.domain.corp.co.in”.

Before the upgrade, the host nqn was “nqn.2014-08.com.vmware:nvme:xxxxxx”. However, following the upgrade, the host nqn transitioned to “nqn.2014-08.in.co.corp.domain:nvme:ABC-DEF-ESX01”. Due to the lack of updates in the initiator section of the storage console reflecting these changes, the ESXi host was unable to establish a session with the storage.

Resolution

Kindly refer to the workaround section of this document for a resolution to the issue

Workaround:

Update the initiators in the storage console with the latest host nqn i.e. from  “nqn.2014-08.com.vmware:nvme:xxxxxx” to latest “nqn.2014-08.in.co.corp.domain:nvme:ABC-DEF-ESX01”

Additional Information

Impact/Risks:

The NVMe namespaces and datastores have become invisible in the vSphere Client