Error: "vmxnet3 tx hang" causing intermittent pod connectivity issues on TKGm Worker Node

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• Intermittent network drops on the pods, leading to connectivity failure between pods running on TKG worker nodes
• Journal logs of the Guest OS have below entries during the issue reported time 
vmxnet3 0000:0b:00.0 eth0: tx hang
vmxnet3 0000:0b:00.0 eth0: resetting
• TKGm worker nodes vMotioned to different ESXi hosts, the issue persists
• Found "Hang detected,numHangQ" entries on the vmkernel logs at ESXi level

Environment

• VMware Tanzu Kubernetes Grid Management 2.5.x

Cause

Issue is not related to TKGm & Photon OS. 
The issue is caused by a defect in the vmxnet3 driver where the transmit queue (tx) becomes unresponsive under specific traffic patterns, leading to a network stall for the affected virtual interface. 

Resolution

Open a case with Broadcom Technical Support with the ESXi host log bundle. Refer KB: Contact Broadcom support