UCS Cisco System see TX hang on VMXNET3 and E1000 within Guest OS

Products

• VMware vSphere ESXi

Issue/Introduction

Issue is related to Guest OS experiencing TX hang when using a UCS chassis and multiple TX queue's with RSS enabled.

Kernel logs will show examples like

• kernel: vmxnet3 0000:0b:00.0 eth1: tx hang
• kernel: vmxnet3 0000:0b:00.0 eth1: resetting

vmkernel will show VM hang detected 

• Vmxnet3: 24934: <VMNAME>,<MAC ADDR>, portID(67108871): Hang detected,numHangQ: 1, enableGen: 0

Note: Tx hang can be due to many different reasons during packet processing and not completing packets in a timely manner. 

Resolution

A TX hang can occur when the incorrect configuration is applied to the network adapter policy of a UCS adapter.


/> get /net/pNics/vmnic5/properties
properties {
   Driver Name:nenic
   Driver Version:1.0.29.0
   Driver Firmware Version:4.3(3c)
   System Device Name:vmnic5

/> get /net/pNics/vmnic5/txqueues/queueCount
8

/> get /net/pNics/vmnic5/stats
tx_queue_[0]_frames_ok: 3750270419
tx_queue_[1]_frames_ok: 0
tx_queue_[2]_frames_ok: 0
tx_queue_[3]_frames_ok: 0
tx_queue_[4]_frames_ok: 0
tx_queue_[5]_frames_ok: 0
tx_queue_[6]_frames_ok: 0
tx_queue_[7]_frames_ok: 0

rx_rss_queue_[0]_frames_ok: 486774846
rx_rss_queue_[1]_frames_ok: 640330374
rx_rss_queue_[2]_frames_ok: 830320848
rx_rss_queue_[3]_frames_ok: 800527260
rx_rss_queue_[4]_frames_ok: 1039549939
rx_rss_queue_[5]_frames_ok: 464512348
rx_rss_queue_[6]_frames_ok: 1108965446

As seen, nenic driver is being used, which is configured for 8 Tx queues but only 1 Tx queue is being used. If RSS is being used it is recommended to use only 1 TX queue configured in the adapter policy. 

If more TX queue is required disable RSS and configure more TX queues with netqueue enabled.

In this instance it is recommended to contact the hardware vendor for network adapter policy when it comes to TX and RX queue settings with RSS/netqueue. 

Please see Cisco Documents for TX queue configuration.
https://www.cisco.com/c/dam/en/us/products/collateral/interfaces-modules/unified-computing-system-adapters/vic-tuning-wp.pdf