VMFS datastore creation on NVMe over TCP fails when using Infinidat arrays

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

Symptoms:

1) VMFS datastore creation fails when attempting to configure it on NVMe over TCP.
2) Logs show connectivity issues or errors related to Infinidat arrays and NVMe over TCP communication.

Environment

VMware vSphere ESXi 8.x

Cause

The direct cause is that the infinidat NVMe/TCP target does not support in-capsule data transfer for IO commands.The below log messages show that the in-capsule data transfer is supported on admin queue (maxCapsuleSize:8256) and not supported on IO queue (maxCapsuleSize:64). 

vmkernel.log

2024-11-06T10:03:40.172Z In(182) vmkernel: cpu48:2099148)nvmetcp:nt_ConnectQueueInt:4737 [ctlr 260, queue 0] connected, qSize:56, maxCapsuleSize:8256, maxH2CData:1048576, hpda:0, cpda:0, hdgst:0, ddgst:0
2024-11-06T10:03:40.225Z In(182) vmkernel: cpu48:2099148)nvmetcp:nt_ConnectQueueInt:4737 [ctlr 260, queue 1] connected, qSize:200, maxCapsuleSize:64, maxH2CData:1048576, hpda:0, cpda:0, hdgst:0, ddgst:0

Resolution

In-capsule data transfer is a feature introduced in ESXi 8.0 P04 to improve performance for small-sized I/O operations. Please check with Infinidat to confirm whether the array supports in-capsule data transfer. If supported, increase the value to a larger setting. If not, revert to the ESXi 8.0 GA build.

Broadcom Engineering team is actively working on a fix.