Vmk interfaces are lost after ESXi reboot/upgrade

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides steps to restore the network on the host when the issue occurs, or to avoid the issue before the ESXi reboot/upgrade.

• After ESXi upgrade from ESXi 6.7 to ESXi 7, all vmk interfaces are lost hence the host cannot be managed by VC, trying to add new vmk interface will fail either.
• The host was prepared for NSX and later uninstalled in a non graceful way . 
• In /var/log/boot or /var/run/log/vmkwarning.log, below logs could be observed:

... cpu95:2097926)WARNING: NetDVS: 2557: Failed to create proxy portset DvsPortset-3, vswitch, 9216

... cpu95:2097926)WARNING: DVSDev: DVSDevConfigInit:149: failed to deserialize config data

Environment

VMware vSphere ESXi 7.0

VMware vSphere ESXi 8.0
VMware NSX

Cause

This issue occurs when there was NSX installed but uninstalled later not in a graceful way, which causes stale NVDS information on the host.

Resolution

Method 1 is easy to perform however all network data will get lost, Method 2 is better only if customer manages the vmk interfaces on standard switches and in such case VDS data could be removed.

Method 1:

In the ESXi DCUI, perform below steps:

1. Network Restore Options -> Restore Network Settings.

2. Configure Management Network, with proper network settings.

Method 2:

1. Find the stale NVDS switches (with attribute "forwarding_class": "vswitch"):

configstorecli config current get -c esx -g network_vds -k switches

2. Get and delete such stale NVDS switches, as below examples:

configstorecli config current get -c esx -g network_vds -k switches -i "2d 0c ## ## ## #1 ## ##-## ## ## ## f4 8b 91 80"
configstorecli config current delete -c esx -g network_vds -k switches -i "2d 0c ## ## ## #1 ## ##-## ## ## ## f4 8b 91 80"

3. Remove the dvsdata.sb

rm /etc/vmware/dvsdata.db

4. (Optional) The vmk interfaces attached to VDS (both non-NSX VDS or NVDS) should be deleted as below

e.g. 

configstorecli config current get -c esx -g network -k vmknics
configstorecli config current delete -c esx -g network -k vmknics -i "vmk10"

5. Reboot the ESXi host

Workaround:

To avoid the issue, please perform below steps before the ESXi reboot/upgrade.

1. find the stale NVDS switch:

net-dvs -l | grep -E '^switch|common.alias'

printout e.g.

switch 2d 0c ## ## ## #1 ## ##-## ## ## ## f4 8b 91 80 (vswitch)

com.vmware.common.alias = testNVDS ,  propType = CONFIG

2. Delete the stale NVDS switch, please change the name "testNVDS" according to previous printout

net-dvs -d testNVDS

 

Additional Information

Impact/Risks:

Network traffic is lost on the host and the host cannot be managed by VC.