UCS Cisco System see TX hang on VMXNET3 and E1000 within Guest OS

Products

• VMware vSphere ESXi

Issue/Introduction

The issue is related to the Guest OS experiencing TX hang when using a UCS chassis and multiple TX queues with RSS enabled.

Kernel logs will show examples like

• kernel: vmxnet3 0000:0b:00.0 eth#: tx hang
• kernel: vmxnet3 0000:0b:00.0 eth#: resetting

var/run/log/vmkernel.log will show VM hang detected 

• Vmxnet3: 24934: <VMNAME>,<MAC ADDR>, portID(6710##71): Hang detected,numHangQ: 1, enableGen: 0

Note: Tx hang can be due to many different reasons during packet processing, and not completing packets in a timely manner. 

Validate via command: Edit the vmnic# as per your environment

vsish -e get /net/pNics/vmnic#/properties
properties {
   Driver Name:nenic
   Driver Version:1.0.##.0
   Driver Firmware Version:4.3(3c)
   System Device Name:vmnic#

vsish -e get /net/pNics/vmnic#/txqueues/queueCount
8

vsish -e get /net/pNics/vmnic#/stats
tx_queue_[0]_frames_ok: 3750270419
tx_queue_[1]_frames_ok: 0
tx_queue_[2]_frames_ok: 0
tx_queue_[3]_frames_ok: 0
tx_queue_[4]_frames_ok: 0
tx_queue_[5]_frames_ok: 0
tx_queue_[6]_frames_ok: 0
tx_queue_[7]_frames_ok: 0

rx_rss_queue_[0]_frames_ok: 486774846
rx_rss_queue_[1]_frames_ok: 640330374
rx_rss_queue_[2]_frames_ok: 830320848
rx_rss_queue_[3]_frames_ok: 800527260
rx_rss_queue_[4]_frames_ok: 1039549939
rx_rss_queue_[5]_frames_ok: 464512348
rx_rss_queue_[6]_frames_ok: 1108965446

Note: As seen, the nenic driver is being used, which is configured for 8 Tx queues, but only 1 Tx queue is being used.

Environment

VMware vSphere ESXi

Cause

A TX hang can occur when the incorrect configuration is applied to the network adapter policy of a UCS adapter.

Resolution

• If RSS is being used, it is recommended to use only 1 TX queue configured in the adapter policy.
• If more TX queue is required, disable RSS and configure more TX queues with netqueue enabled.

In this instance, it is recommended to contact the hardware vendor for network adapter policy when it comes to TX and RX queue settings with RSS/netqueue.