VM to VM communication issues when using NSX-T with ESXio/DPUs enabled.

Products

• VMware NSX

Issue/Introduction

• Customer reports that the virtual desktop screen freezes every few seconds to every few minutes
• VDI Machines are unable to reach application servers.  Further investigation shows packets are not always routing as expected in the environment.  IE, response packets leave a different uplink/TEP than the initial received packet, and take a different network path when investigated using TraceFlow in the NSX-T UI.

Environment

• Impacted Version : NSX 4.0, NSX 4.1
• NSX-T installed on ESXIO SmartNICs (DPUs)

Cause

1.) UDP based applications will experience packet drops due to flow invalidation.

2.) When a Flow is invalidated then we observed that few packets are looped back from the DPU. These looped back packets are dropped since these packets are never expected to be received from the uplink port.

Resolution

Workaround: disable hardware offloading on the DPU.

• To disable offload:  nsxdp-cli ens fpo offload-flows set --enable --dvs-alias [DVS_ALIAS]
• Confirm hardware offload status:  nsxdp-cli ens fpo offload-flows get --dvs-alias [DVS_ALIAS]