ESXi host lost network connectivity during preparation for NSX / ESXi Host Unresponsiveness during NSX Preparation

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

ESXi host was prepared for NSX in the past.
Host is being re-prepared for NSX.
Host's management vmknic interface is on the switch used for NSX.
During host preparation, host will become disconnected from the vCenter, and NSX.
Host's management vmknic is not pingable.

/var/log/vmkernel.log may contain logs similar to the logs below:

YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)ENS: Ens_VdsPropUnregister:586: Unregistering VDS Prop CBs for ps DvsPortset-0
YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 534: DvsPortset-0: failed to swap from type cswitch to type vswitch reverting to cswitch
..
YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)NetDVS: 187: A client exists for data com.vmware.nsx.kcp.enable on portset DvsPortset-0.
..
YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: kcp: KCP_DvpgPropDeactivate:163: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Failed to unregister property: com.vmware.nsx.kcp.enable for DvsPortset-0 : Busy
YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)NetDVS: 187: A client exists for data com.vmware.nsx.kcp.enable on portset DvsPortset-0.
YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: kcp: KCP_VdsPropActivate:283: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Failed to register property: com.vmware.nsx.kcp.enable for DvsPortset-0 : Already exists
..
YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)vswitch: VSwitchActivate:11778: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Failed to post netevent com.vmware.nsx.vswitch.activate.24476730 on ps DvsPortset-0: Failure
YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 309: Failed to swap in type vswitch
..
YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 534: DvsPortset-0: failed to swap from type cswitch to type vswitch reverting to cswitch
..
YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)cswitch: VSwitchActivate:10077: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Failed to post NSX_CSWITCH_ACTIVATE on ps DvsPortset-0: Failure
YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 309: Failed to swap in type cswitch
..
YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 540: DvsPortset-0: failed to revert to type cswitch, reverting to nulldev

Output of "esxcfg-vswitch -l" run on the impacted host returns error similar to the error below:

#esxcfg-vswitch -l
DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks
Listing failed for DVSwitch: DvsPortset-0, Error: Unable to get the dvs name: Status(bad0007)= Bad parameter

Cause

This issue is caused by stale NSX properties left in ESXi host's distributed switch configuration (e.g. kcp), which will cause a conflict when the host is being re-prepared for NSX. Due to this conflict, ESXi is unable to install the properties and set a correct switch type, which will result in switch type failing to nulldev device.

Resolution

This issue is resolved in VMware NSX 4.2.2 available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

To recover connectivity of the ESXi host once the issue has occurred, host's management network needs to re-connected (usually via standard switch), or host's network stack needs to be rebuild.

To prevent this issue, if the ESXi host was previously prepared for NSX and is being prepared for NSX again, you can confirm configuration of the distributed switch and remove the stale entries before re-preparing the host for NSX by running the NSX cleanup script as per NSX VIBs are not removed from an ESXi host.

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.

Additional Information

Understanding network rollback and recovery in vSphere 7.0 and later

Configuring Standard vSwitch (vSS) or virtual Distributed Switch (vDS) from the command line in ESXi

Recover ESXi host connectivity when management is on DVS

Also see KB article In an NSX Security Only Install, configuring ESXi hosts for NSX might fail while applying the host switch configuration. and NSX Host preparation fails with an "Host configuration: Found security-only non-CVDS on the host" message.