Host ESXi networking missing after host upgraded to 7.0 when NSX-T VIBS were removed manually

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

Symptoms:
ESXi host is upgraded to ESXI 7. After reboot, all host networking is missing. DCUI screen is blank:

Customize System (F2) shows all host networking options are grayed out:

Enable ESXi Shell and log in as root and networking commands fail.
  NOTE: net-dvs sometimes runs and sometimes fails as below

  net-dvs -l    returns failed to get config data: Not Initialized
  esxcli network ip interface list   returns  "Unable to get node: Not found"
  esxcfg-vmknic -l returns nothing
3-networking -commands-fail.png

If the N-VDS was removed manually, you may receive the error:

Configuration Upgrade Failure
Please reboot to rollback to the older version
Failed Modules:
/usr/lib/vmware/configmanager/upgrade/lib/libupgradeddvsconfig.so

Environment

VMware NSX-T
VMware vSphere ESXi 7.x

Cause

The host was previously used for NSX-T and vibs were manually removed via esxcli software vib remove -n <vibs name>

This is an incorrect uninstall method and leaves stale objects on the host, including the N-VDS. The presence of the N-VDS prevents host networking from initializing and must be removed.

Resolution

This is a known issue that requires the workaround in this article

Workaround:
You can either reinstall ESXi host from scratch or create a new vSwitch and management vmkernel

Sometimes creating a new vSwitch will not work

See KB Configuring vSwitch or vNetwork Distributed Switch from the command line in ESXi/ESX for information steps on creating the vSwitch via the CLI

Process to remove the stale N-VDS before upgrading
The below process has been confirmed in VMware lab environment to be non-disruptive.2

However, VMware always recommends that customers make changes of this nature during a maintenance window.3

Additionally VMware recommends that customers always ensure full backups prior to any maintenance window.4

1. SSH into each ESXi host

2. Search for a stale N-VDS by checking for the name of the orphaned NSX DVS

net-dvs -l | less

Note: you may have to go down many pages to get to the NSX DVS switch details

Look for the following output - Note: This Example is the NSX DVS configuration as seen from support bundles

-----SNIP----
switch cb cf 7b 58 37 50 43 b2-8b f1 95 b3 88 0f 0b a9 (vswitch)
        max ports: 10752
        global properties:
                com.vmware.common.opaqueDvs = true,    propType = CONFIG
                com.vmware.common.alias = nvds-overlay-acdc,   propType = CONFIG
                com.vmware.common.uplinkPorts:
                        uplink-1, uplink-2
                        propType = CONFIG
                com.vmware.common.portset.mtu = 9000 , propType = CONFIG
                com.vmware.etherswitch.cdp = LLDP, listen
                        propType = CONFIG
                com.vmware.common.respools.version = version3 ,         propType = CONFIG
---END SNIP---

3. Remove the orphaned NSX DVS

net-dvs -d nvds-overlay-acdc
net-dvs --persist <----- if you forget this step, the stale nvds will reappear after reboot!!!!

4. Check that the orphaned NSX DVS is now removed

net-dvs -l | grep -i nsx

5. Remove stale entries from /etc/vmware/esx.conf
a. Travel to /etc/vmware
cd /etc/vmware
b. Backup the config file
cp esx.conf esx.conf.bkup
c. Edit the config file
vi esx.conf
d. Find the stale config lines and delete them by pressing "dd"
search for
/net/dvswitch/child[xxxx]/dvsClassName = "vswitch"

NOTE: xxxx will be a four digit number. Remove all the lines that have this same number

ii. Sample lines. ALL of these lines need to be removed
/net/dvswitch/child[0001]/uplinks/child[0000]/connectionId = "0"
/net/dvswitch/child[0001]/uplinks/child[0000]/dvpId = "up1"
/net/dvswitch/child[0001]/uplinks/child[0000]/pnic = "vmnic8"
/net/dvswitch/child[0001]/uplinks/child[0001]/pnic = "vmnic9"
/net/dvswitch/child[0001]/uplinks/child[0001]/dvpId = "up2"
/net/dvswitch/child[0001]/uplinks/child[0001]/connectionId = "0"
/net/dvswitch/child[0001]/name = "DvsPortset-1"
/net/dvswitch/child[0001]/numPorts = "64"
/net/dvswitch/child[0001]/dvsClassName = "vswitch"
iii. save the file. ":x" or ":wq" will save the file in vi
iv. run auto_backup.sh to save the changes to the bootbank

6. Repeat step 4 & 5 on all other hosts in the cluster

Note1: These steps should not be executed without VMware Support actively engaged

Note2: The steps included in the above process are non-disruptive in lab environments. VMware always recommends making changes like this under a maintenance window and any business change controls already in effect, to ensure that unexpected events do not have a unexpected impact on production.

Note3: VMware recommends full backups prior to any maintenance activity

Additional Information

Impact/Risks:
Host ESXi networking is missing after host upgrade