Degraded Performance Observed with Cisco Native FNIC driver (NFNIC) on ESXi 6.7

Products

• VMware vSphere ESXi

Issue/Introduction

1. A reduction in storage performance (higher latency and/or reduced throughput) has been observed on Cisco UCS blades running ESXi 6.7
2. Higher Kernel Average (KAVG) latency measured in ESXTOP ( > 1ms)
3. Reduced throughput (upwards of 50% reduction in total throughput) 
Note: Environments using the legacy vmklinux Cisco FNIC driver on ESXi 6.7 do not notice any reduction in performance or increased latency

Environment

VMware vSphere ESXi 6.7

Cause

The Native FNIC driver for ESXi 6.7 did not have Physical Address Extension (PAE) flag enabled for the driver. This caused the VMkernel to have to split commands before handing them over to the Cisco driver. As a result, this caused the Kernel Average (KAVG) to increase as the commands were being processed. This behavior also caused throughput to be impacted (upwards of a reduction of 50%)

Resolution

Cisco has released a new version of the Native FNIC driver (nfnic version 4.0.0.52) for ESXi 6.7 that has Physical Address Extension (PAE) enabled. Here is the link to download the newest NFNIC driver from Cisco:

VMware vSphere - Essentials 6.7 and search for "4.0.0.52"

Cisco Article: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt45251