ESXi vmnic Network Instability with Intel E810-C (icen Driver)

Products

• VMware vSphere ESXi

Issue/Introduction

• VM randomly becomes unreachable on a particular host.
  
  
• Physical NIC driver in use is Intel E810-C for SFP.
  
  
• vmnic UP/down events are observed in /var/log/vobd.log.
  
  
• In ESXi logs, under /var/log/vmkwarning.log observed repeated HClk Watchdog resets for an icen pNIC.
  
  

  WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7044: vmnic# : scheduler(0x432333730bd0)/device(0x430908433fc0) 0/1 lock up [stopped=0]:
  WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7052: detected at 2921####765 while last xmit at 2921#####67 and 432561 bytes in flight [window 432432 bytes]
  WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7059: and last enqueued/dequeued at 2921#####61/2921#####70 [stress 0]
  WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7064: with 2859 pkts inflight
  WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld:7099: vmnic#: packets completion seems stuck, issuing reset
  WARNING: World: vm 16490####: 43##: vm not found
  WARNING: icen: indrv_CleanRxQueueBuffers:559: rxQueue->rxBufCB is a NULL pointer!
  WARNING: icen: indrv_AllocTxBuf:253: 0000:##:00.2: Failed to allocate heap for TX queue 87 buffer
  WARNING: icen: indrv_AllocRssVsisInHw:10406: 0000:##:00.2: VSI 7: Failed to allocate the queues for NetQueue RSS, Status: Failure
  WARNING: icen: icen_UnsetRxQueueFilter:8574: 0000:##:00.2: VSI 0: Failed to remove cloud filter ID 0 on queue 0, Status: Failure, AQ Error: ICE_AQ_RC_EACCES

   

• Host /var/log/vmkernel.log , multiple entries of heap allocation error is observed. 
  
  

  WARNING: icen: icen_CleanControlQ:6817: 0000:##:00.2: Failed to allocate heap for the Admin queue event

Environment

vSphere 8.x 

Cause

The errors of heap allocation is caused due to a known bug where icen driver is unable to release heap memory properly leading to heap memory exhaustion introduced in async icen 1.14.x.

Resolution

 Upgrade icen driver to version 2.2.2.0 or later.

Additional Information

VMs intermittently lose network connectivity on hosts with the ICEN driver and EDP standard mode enabled 

vmnic down / up alerts in cluster with message "packets completion seems stuck, issuing reset" in logs