Load balancing is not performed by the ESXi hosts for NVMe over TCP devices

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• ESXi hosts are failing to distribute I/O traffic across available NVMe over TCP controllers. Instead, all traffic is traversing a single controller, leaving other available path idle.
  
  
• HBA statistics can be verified using the command: esxcli storage core adapter stats get.
  

  vmhbaX:
   Successful Commands: 3981354
   Blocks Read: 341188251
   Blocks Written: 167846544
   Read Operations: 2667435
   Write Operations: 1310652
   Reserve Operations: 0
   Reservation Conflicts: 0
   Failed Commands: 0
   Failed Blocks Read: 0
   Failed Blocks Written: 0
   Failed Read Operations: 0
   Failed Write Operations: 0
   Failed Reserve Operations: 0
   Total Splits: 0
   PAE Commands: 0

  vmhbaY:
   Successful Commands: 0
   Blocks Read: 0
   Blocks Written: 0
   Read Operations: 0
   Write Operations: 0
   Reserve Operations: 0
   Reservation Conflicts: 0
   Failed Commands: 0
   Failed Blocks Read: 0
   Failed Blocks Written: 0
   Failed Read Operations: 0
   Failed Write Operations: 0
   Failed Reserve Operations: 0
   Total Splits: 0
   PAE Commands: 0

Environment

VMware ESXi 8.0.x

Cause

• This behavior typically occurs because one or more controller Path State is reported as active unoptimized. In this configuration, the host treats the path as standby, only directing I/O to it if the optimized (active) path fails.
  
  
• To confirm the current path states claimed by the HPP, run: esxcli storage hpp path list
  
  Sample Output:
  [root@esx##### :~ ] esxcli storage hpp path list
tcp.vmnicA:##:##:##:##:##:##-tcp.XXX.XX.XX.XXX:4420-eui. ###################### 
Runtime Name: vmhbaX:C#:T#:L#
Device: eui. ###################### 
Device Display Name: NVMe TCP Disk (eui. ###################### )
Path State: active
Path Config: {ANA GRP_id=2,ANA GRP_state=AO, health=UP}

tcp.vmnicB:##:##:##:##:##:##-tcp.XXX.XX.XX.XXX:4420-eui. ###################### 
Runtime Name: vmhbaY:C#:T#:L#
Device: eui. ###################### 
Device Display Name: NVMe TCP Disk (eui. ###################### )
Path State: active unoptimized
Path Config: {ANA GRP id=2,ANA GRP state=ANO, health=UP}

Resolution

Contact your storage vendor to review the array-side configuration. Ensure that all intended paths are configured as active-active.