NVMe over TCP devices encounter frequent path loss and performance degradation when data digest is enabled.

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

VMFS datastores backed by NVMe over TCP storage devices may have frequent path loss events occur which can result in performance degradation when data digest feature is enabled.

Alerts for path redundancy loss may be frequently triggered in the vSphere client and vobd.log may contain log-lines similar to the following:

[psastorCorrelator] ###########us: [esx.problem.storage.redundancy.lost] Lost path redundancy to storage device eui.################################. Path vmhbaXX:C0:T0:LX is down. Affected datastores: Unknown.
vobd[########]:  [psastorCorrelator] ###########us: [vob.psastor.psastorpath.pathstate.dead] 

Environment

ESXi 8.X

Cause

While write IOs are in transit inside ESXi, the Guest-OS/Application issuing the IO can modify the IO e.g. before the IO has been completed by the nvmetcp driver.

If data digest is enabled then if data is modified after the digest checksum is calculated (inside the nvmetcp driver) then it will cause digest mismatch on the target side resulting in the controller dropping the connection and the host having to repeatedly re-establish the connection.

Fix for this issue will be included in a future nvmetcp driver version included in ESXi 8.X and ESXi 9.X

Resolution

Disable data digest feature, header digest is not affected by this issue and can be left enabled.

Data digest is disabled by default - if the default configuration (both data digest and header digest disabled) is used then this issue cannot occur.