ESXi host lost network connectivity during preparation for NSX / ESXi Host Unresponsiveness during NSX Preparation
search cancel

ESXi host lost network connectivity during preparation for NSX / ESXi Host Unresponsiveness during NSX Preparation

book

Article ID: 392477

calendar_today

Updated On:

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

  • ESXi host was prepared for NSX in the past. 
  • Host is being re-prepared for NSX.
  • Host's management vmknic interface is on the switch used for NSX.
  • During host preparation, host will become disconnected from the vCenter, and NSX. 
  • Host's management vmknic is not pingable.
  • /var/log/vmkernel.log may contain logs similar to the logs below:

    YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)ENS: Ens_VdsPropUnregister:586: Unregistering VDS Prop CBs for ps DvsPortset-0
    YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 534: DvsPortset-0: failed to swap from type cswitch to type vswitch reverting to cswitch
    ..
    YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)NetDVS: 187: A client exists for data com.vmware.nsx.kcp.enable on portset DvsPortset-0.
    ..
    YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: kcp: KCP_DvpgPropDeactivate:163: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Failed to unregister property: com.vmware.nsx.kcp.enable for DvsPortset-0 : Busy
    YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)NetDVS: 187: A client exists for data com.vmware.nsx.kcp.enable on portset DvsPortset-0.
    YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: kcp: KCP_VdsPropActivate:283: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Failed to register property: com.vmware.nsx.kcp.enable for DvsPortset-0 : Already exists
    ..
    YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)vswitch: VSwitchActivate:11778: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Failed to post netevent com.vmware.nsx.vswitch.activate.24476730 on ps DvsPortset-0: Failure
    YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 309: Failed to swap in type vswitch
    ..
    YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 534: DvsPortset-0: failed to swap from type cswitch to type vswitch reverting to cswitch
    ..
    YYYY-MM-DD In(182) vmkernel: cpu69:2103540 opID=####985d)cswitch: VSwitchActivate:10077: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Failed to post NSX_CSWITCH_ACTIVATE on ps DvsPortset-0: Failure
    YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 309: Failed to swap in type cswitch
    ..
    YYYY-MM-DD Wa(180) vmkwarning: cpu69:2103540 opID=####985d)WARNING: NetHotswap: 540: DvsPortset-0: failed to revert to type cswitch, reverting to nulldev

  • Output of "esxcfg-vswitch -l" run on the impacted host returns error similar to the error below:

    #esxcfg-vswitch -l
    DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks
    Listing failed for DVSwitch: DvsPortset-0, Error: Unable to get the dvs name: Status(bad0007)= Bad parameter

Environment

VMware ESXi

VMware NSX-T Data Center

VMware NSX

Cause

This issue is caused by stale NSX properties left in ESXi host's distributed switch configuration (e.g. kcp), which will cause a conflict when the host is being re-prepared for NSX. Due to this conflict, ESXi is unable to install the properties and set a correct switch type, which will result in switch type failing to nulldev device. 

Resolution

This issue is resolved in VMware NSX 4.2.2 available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

To recover connectivity of the ESXi host once the issue has occurred, host's management network needs to re-connected (usually via standard switch), or host's network stack needs to be rebuild. 

To prevent this issue, if the ESXi host was previously prepared for NSX and is being prepared for NSX again, you can confirm configuration of the distributed switch and remove the stale entries before re-preparing the host for NSX:

  1. SSH to the ESXi host. 
  2. Confirm stale properties are present on the switch (which is currently not being used by NSX):
    # net-dvs -l | grep "common.alias\|kcp.enable\|vdsSecurity.enabled"
  3. If the properties in the output are set to "true" (sample below), proceed to the next step:
                    com.vmware.nsx.kcp.enable = true ,      propType = CONFIG
  4. To disable these properties:
    # net-dvs -u com.vmware.nsx.kcp.enable -p hostPropList <dvsName>
    # net-dvs -u com.vmware.nsx.vdsSecurity.enabled -p hostPropList <dvsname>
    Note the <dvsName> is in output of command run in step b. (common.alias).
  5. The host is now ready to be prepared for NSX. 

 

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.

Additional Information

Powered by Wolken Software