ESXi hosts fail to connect to additional NVMe over TCP controllers

Products

• VMware vSphere ESX 8.x
• VMware vSphere ESXi

Issue/Introduction

• ESXi hosts have access to a number of NVMe contrrollers via NVMe over TCP.
  
  
• All hosts share a similar NVMe configuration.
  
  
• ESXi host access to additional NVMe controllers is configured
  
  
• On most hosts, the ESXi hosts successfully connect to the newly configured controllers.
  
  
• On one or more hosts, connection to the new controllers fails.
  
  
• An error is reported in the vSphere Client:
  
  An error occurred during host configuration: Unable to connect to the specified transport target using the provided connection specification.    

Cannot change the host configuration. Failed to connect to NVMe subsystem '###################' at '\####:####@#####'

Environment

VMware vSphere ESXi 8.x

Cause

This arises if the maximum number of connections to the storage subsystem is exceeded. 

While storage IPs are reachable from the ESXi host, on host boot /var/log/vmkernel.log will report logging similar to:
vmkernel: cpu32:2098675)NVMFDEV:2678 controller ###, queue 0
vmkernel: cpu32:2098675)NVMFDEV:2708 controller ###, queue 0, connecting with SQ flow control enabled
vmkernel: cpu1:2098781)nvmetcp:nt_ReceiveFromSocket:1358 [ctlr ###, queue 0] failed to read socket, expected 8, received 0.: Broken pipe
vmkernel: cpu1:2098781)nvmetcp:nt_ReceivePduHeader:3264 [ctlr ###, queue 0] failed to receive common header: Broken pipe
vmkernel: cpu1:2098781)nvmetcp:nt_RxWorldFunc:3537 [ctlr ###, queue 0] failed to receive rxPdu, state 1, fes 0, fei 0: Broken pipe
vmkernel: cpu1:2098781)nvmetcp:nt_RxWorldFunc:3566 [ctlr ###, queue 0] Failed to receive rxPdu: Broken pipe
vmkwarning: cpu33:2098675)WARNING: NVMFDEV:1141 Failed to fabric connect controller ###, status: Failure

/var/log/vmkernel.log may also report repeated logging for the additional controllers similar to:
vmkernel: cpu2:33514708)nvmetcp:nt_ReleaseQueueAndWait:213 [ctlr ###, queue 4] Waiting for queue refCount(1) to be 0: Timeout; time elapsed 619480000 ms

 

Resolution

Work with the storage vendor to ensure that the maximum number of connections to the storage subsystem is not exceeded.