NVMe datastores disappear post ESXi FQDN change

Products

• VMware vSphere ESXi

Issue/Introduction

• Datastores become inaccessible or disappear at the vSphere level after an ESXi hostname change.
• These issues can also occur later if the ESXi host was not rebooted or upgraded following a Fully Qualified Domain Name (FQDN) change
• /var/log/vmkernel.log may have the below messages:

[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:177 vmkParams.hostid:  ########-####-####-####-########6410
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:178 vmkParams.hostnqn: nqn.2014-08.in.xxxx.####.####:nvme:####-ESX01
[YYYY-MM-DDTHH:MM:SS] cpu1:2098192)WARNING: NVMFDEV:882 Failed to transport connect controller 257, status: Failure
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:179 vmkParams.subnqn:  nqn.1988-11.com.dell:powerstore:00:###########
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:190 vmkParams.trType:  FC
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:192 vmkParams.trwwnn:  ############
[YYYY-MM-DDTHH:MM:SS] cpu1:2098192)NVMEDEV:1565 Ctlr 257 freeing
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)NVMFDEV:194 vmkParams.trwwpn:  ############
[YYYY-MM-DDTHH:MM:SS] cpu1:2098192)NVMEDEV:9067 Cancel requests of controller 257, 0 left.
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)WARNING: lpfc: lpfc_remoteport_add:1875: 0:6067 No match ############
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)WARNING: lpfc: lpfcGetRemotePort:2491: 0:1073 Get Remote Port Failed
[YYYY-MM-DDTHH:MM:SS] cpu26:2098194)NVMEDEV:1393 Ctlr 259 allocated, maximum queues 16
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)WARNING: lpfc: lpfcConnectCtrlr:2543: 0:1076 Failed in Remote Port Get WWNN : ########### WWPN : ###########
[YYYY-MM-DDTHH:MM:SS] cpu12:2098193)WARNING: NVMFDEV:882 Failed to transport connect controller 258, status: Failure
[YYYY-MM-DDTHH:MM:SS] cpu1:2098192)WARNING: NVMFEVT:1084 Failed to connect controller nqn.1988-11.com.####:########:00:#############, status: Failure

 

Environment

• vSphere ESXi 8.X
• VCF 9.0
• vSphere ESXi 9.0

Cause

• ESXi establishes a storage connection via NQN to retrieve namespace information from the storage array. If the ESXi hostname changes and the host is rebooted, the NQN controller names between ESXi and the storage array will also change.
• This results in ESXi being unable to fetch the namespace, causing the datastores to become unavailable, even though the controller shows as online.

Resolution

To verify if the hostname on the ESXi host has changed, compare the Host NQN with the NQN configured on the storage array.

1. Use the following command on the ESXi host to retrieve the Host NQN:
   
   # esxcli nvme info get
   
   
2. If the Host NQN differs from the one configured on the storage array, update the initiators in the storage console with the latest Host NQN
3. Once the NQN has been updated, perform a storage rescan on the ESXi host. The datastores should become visible after the rescan.

Additional Information

HOST NQN not updating due to which LUNs connectivity impacted