ixgbe driver reload during NSX installation

Products

• VMware NSX

Issue/Introduction

5-10 second network outage at the end of NSX host prep on servers which only has ixgbe driver based physical NICs.

Environment

VMware NSX Data Center for vSphere 6.4.x

Cause

The ixgbe driver will be reloaded with Receive Side Scaling (RSS) option to improve the vxlan throughput. This typically takes about 5-10 seconds. Very rarely (if the system memory is very badly fragmented), the driver reload may fail.

Note: The ixgben (native) driver does not experience this issue.

Resolution

This is a known issue affecting VMware NSX for vSphere 6.4.x.


Workaround:
To work around this issue, apply either one of these workarounds:

Workaround #1

1. In the vSphere Web Client, click Policies and Profiles.
2. Click Host Profiles.
3. Click Edit Host Profile and expand the General System Settings folder.
4. Expand Kernel Module Configuration, then select Kernel Module.
5. If ixgbe is loaded, select it.
6. Select Kernel Module Parameter > max_vfs . If the parameter Value is non-empty, click Cancel.
7. Select Kernel Module Parameter > CNA . If the parameter Value is non-empty, click Cancel.
8. Select Kernel Module Parameter > RSS . If the parameter Value is non-empty, click Cancel.
9. Set RSS value to 4,4,4,4,4,4,4,4.
10. Click Finish.

Notes:

• If you reached step 10 above, reboot the host after moving all the virtual machines out. If not, no need to reboot.
• Then prepare the host for NSX where the VIBs are pushed to the hosts.

Workaround #2

Manual Install using the "Resolve" button from Host Preparation Tab.

Here are the steps required to get a cluster prepared, these steps assume the issue was found during host prep, and you have a partially prepared cluster.

1. Make a note of what ESXi your NSX Manager is on.
2. Make sure NSX Manager is on an already prepared ESXi, NSX Manager can also be on an ESXi in a different cluster that is not being prepared for NSX.
3. From Network and Security > Installation and Upgrade -> Host Preparation tab, on the cluster you started prepping, select a host that has not been prepped yet. There are two options on doing this:

Option #1 - Resolve

1. Do an individual "Resolve" on the ESXi host. This has the host reach out to the NSX Manager to get the appropriate VIB for the host version, download and install it.
2. Monitor the ESXi host. The network disconnects from 5-10 seconds but should return to normal. If it does, the ESXi host requires a reboot.
3. Repeat this process for the other ESXi hosts in the cluster until the entire cluster is prepared for NSX.

Option #2 - Manual Install

1. Do a manual VIB install on an ESXi host. For more information, see the Host Preparation section of the NSX Troubleshooting Guide.
2. Navigate to https://NSXMGR_IP/bin/vdn/vibs-6.4.0/6.0-7564187/vxlan.zip.
3. You need the VIB for your ESXi host version. The VIB to look for is the "esx-nsxv" and load it to the shared storage where all affected ESXi host have access to.
4. From Hosts and Clusters > Select your Cluster > Select your Host > Enter the Host into maintenance mode.
5. SSH to the ESXi host as the root user.
6. Validate that the VIBs are installed by running this command:
   
   esxcli software vib list | grep nsx
   
   Note: This should not return anything.
    
7. Install the VIB manually by running this command:
   
   esxcli software vib install -d /filepath/vxlan.zip
    
8. Since the ESXi host was placed in Maintenance Mode, there should not be any network disruption.
9. Exit the ESXi host out of Maintenance Mode and validate if the VIBs are installed correctly by running this command:
   
   esxcli software vib list | grep nsx
   
   Note: This should return a value with the esx-nsxv VIB.
    
10. Repeat all steps from Option #2 for the other ESXi in the cluster.

Workaround #3

Move to the ixgben driver on the ESXi host which does not have strict physical requirements.