Intermittent Packet Loss on Uplink for an NSX Environment

Products

• VMware
• VMware NSX Networking
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Small percentage of packets being dropped intermittently in the NSX environment.
• ESXi(s) are installed on Cisco UCS with enic Driver.
• From the Packet Capture, it shows that the packet drops are seen on the physical NIC of the ESXi after the traffic is sent out form the VM/DLR towards the physical NIC of the ESXi.

Cause:

By default on the Cisco UCS that are using enic driver, the RSS is disabled which will result in having only one queue on the Physical NIC for the receiving path. As the amount of VXLAN traffic is normally higher than the normal, you may see this issue on the VXLAN traffic and not the normal traffic.

Environment

VMware vSphere ESXi 6.0
VMware vCenter Server 6.0.x
VMware vCenter Converter Standalone 6.x
VMware NSX for vSphere 6.1.x
VMware Update Manager 6.0
VMware NSX for vSphere 6.3.x
VMware vCenter Server Appliance 6.0.x
VMware NSX for vSphere 6.2.x

Resolution

To resolve the issue, raise a service request with your hardware vendor to implement below suggested recommendations as per their best practices.

• RSS Enabled/Disabled
• Number of queue and the buffer size for each one while not encountering memory overload.
• VXLAN-Offloading.
• TSO Enable/Disable.