Logical switches and segments still present in the vCenter Server after successful uninstall of transport nodes
search cancel

Logical switches and segments still present in the vCenter Server after successful uninstall of transport nodes

book

Article ID: 303454

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:

  • Logical Switches and segments still present in the vCenter Server after the associated NSX transport node is uninstalled.
  • After the hosts are removed from NSX-T, there are some opaque networks showing in vCenter.
  • In the example below, under Cluster, Summary in the vSphere client, the logical switch LS-2 is still present after removing NSX-T components form hosts:

  • In the /var/log/hostd.log file, you see entries similar to:

    2019-10-15T13:22:20.251Z info hostd[10703B70] [Originator@6876 sub=Hostsvc.NetworkProvider] GetDvsById: dvs 92 cb 71 64 3d 6e 47 6d-9b 49 7d 86 95 44 a5 f3 not found
    2019-10-15T13:22:20.251Z warning hostd[10703B70] [Originator@6876 sub=Hostsvc.NetworkProvider] Error fetching Dvs with id 92 cb 71 64 3d 6e 47 6d-9b 49 7d 86 95 44 a5 f3 not found, may be deleted. Ignoring exception
    2019-10-15T13:22:20.259Z verbose hostd[FD80B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: config.network, ha-host. Sent notification immediately.

    2019-10-15T13:22:20.288Z info hostd[10703B70] [Originator@6876 sub=Hostsvc.NetworkProvider] GetDvsById: dvs 92 cb 71 64 3d 6e 47 6d-9b 49 7d 86 95 44 a5 f3 not found
    2019-10-15T13:22:20.288Z warning hostd[10703B70] [Originator@6876 sub=Hostsvc.NetworkProvider] Error fetching Dvs with id 92 cb 71 64 3d 6e 47 6d-9b 49 7d 86 95 44 a5 f3 not found, may be deleted. Ignoring exception
    2019-10-15T13:22:20.288Z verbose hostd[10140B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: networkInfo, networkSystem. Applied change to temp map.
    2019-10-15T13:22:20.288Z verbose hostd[10140B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: networkConfig, networkSystem. Applied change to temp map.

    2019-10-15T13:22:42.398Z error hostd[10980B70] [Originator@6876 sub=Hostsvc] IPC recv failed: the peer has performed an orderly shutdown , expecting 284 received 0
    2019-10-15T13:22:42.398Z info hostd[10980B70] [Originator@6876 sub=Hostsvc] IPC close socket
    2019-10-15T13:22:42.398Z error hostd[10980B70] [Originator@6876 sub=Hostsvc] OpaqueNetworkIPC failed to receive msg
    2019-10-15T13:22:42.398Z info hostd[10980B70] [Originator@6876 sub=Hostsvc] IPC close socket
    2019-10-15T13:22:42.398Z info hostd[10980B70] [Originator@6876 sub=Hostsvc] OpaqueNetworkMgr: Stopped OpaqueNetwork-IPC thread
    2019-10-15T13:22:42.398Z info hostd[10980B70] [Originator@6876 sub=ThreadPool] Thread delisted
     
  • In the /var/log/vmkernel.log file, you see entries similar to:

    2019-10-15T13:22:20.141Z cpu0:337638)FlowCache.fc: FC_DispatchDataSlabDestroy:127: [nsx@6876 comp="nsx-esx"]Dispatch data slab destroy
    2019-10-15T13:22:20.141Z cpu0:337638)FlowCache.fc: FC_MemCleanup:343: [nsx@6876 comp="nsx-esx"]Cleanup all Slabs
    2019-10-15T13:22:20.141Z cpu0:337638)FlowCache.fc: FCCacheResourcesDestroy:3072: [nsx@6876 comp="nsx-esx"]Flow Cache Resources destroyed
    2019-10-15T13:22:20.330Z cpu10:340002)netschedHClk: NetSchedHClkWatchdogSysWorld:4352: vmnic2: watchdog failed to acquire the device port 0x400000a while hclk object reference count is 0: Not found
    2019-10-15T13:22:20.330Z cpu10:340002)netschedHClk: NetSchedHClkWatchdogSysWorld:4364: vmnic2: hclk scheduler instance clean up
    2019-10-15T13:22:20.331Z cpu15:65621)NetPort: 1881: disabled port 0x8
    2019-10-15T13:22:42.398Z cpu2:339707)WARNING: UserObj: 5436: nsx-vim: Unimplemented operation on 0x4395843cd170/SOCKET_UNIX_SERVER
    2019-10-15T13:22:52.437Z cpu13:338345)dvfilterUser DVFILTERUSER_shm_pool_destroy:364: Destroying dvfilter shared memory pool  ctrl 0x430eae1073d0, data 0x430eae107430
    2019-10-15T13:22:52.437Z cpu13:338345)nsxt-dvfilterUser: DU_DestroyShm:188: Destroying shmData 0x430eae1073d0
    2019-10-15T13:22:52.437Z cpu13:338345)nsxt-dvfilterUser: DU_DestroyShm:188: Destroying shmData 0x430eae107430
    2019-10-15T13:22:52.437Z cpu13:338345)dvfilterUser DVFILTERUSER_shm_pool_destroy:364: Destroying dvfilter shared memory pool  ctrl 0x430eae10f630, data 0x430eae10f6d0
    2019-10-15T13:22:52.437Z cpu13:338345)nsxt-dvfilterUser: DU_DestroyShm:188: Destroying shmData 0x430eae10f630
    2019-10-15T13:22:52.437Z cpu13:338345)nsxt-dvfilterUser: DU_DestroyShm:188: Destroying shmData 0x430eae10f6d0

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.



Cause

This issue occurs because the LogicalSwitchBulkMsg during transport node deletion did not reach the transport node or was not sent by management plane to transport node.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.0 and later, available at Broadcom Downloads.

Workaround:

If the segments were present and persist after upgrade, file a support request with VMware Support and quote this Knowledge Base article ID (76521)  in the problem description. For more information, see How to Submit a Support Request.

Additional Information

The information about opaque networks are communicated to the vCenter Server from the ESXi hosts, so it must be the host who still have them in the case that they were not deleted correctly.

To check:

  1. Navigate to the vCenter mob, (https://[fqdn of vcenter]/mob), select the host, then config, then network.
  2. In case that the host does not have opaque networks the field that is showing must report 'null' in the right.
  3. If you see the hyperlink 'opaqueNetwork' in the far right as in the screenshot below (so you can click and see the logical switch portgroups), then the host is advertising this portgroups to vCenter:

A similar check can be done in the host CLI looking at the proxy switch in there, if there are UUIDs on the property property ‘com.vmware.port.extraConfig.opaqueNetwork.id’, then you have still an N-VDS proxy switch, and logical switches in there:

For example:

[root@esxi:~] net-dvs -l | grep '.opaqueNetwork.id'

                com.vmware.port.extraConfig.opaqueNetwork.id =  ,       propType = CONFIG
                com.vmware.port.extraConfig.opaqueNetwork.id =  ,       propType = CONFIG
                com.vmware.port.extraConfig.opaqueNetwork.id = 624ea4d1-ba56-4805-b244-ecc95ca0c2a1 ,   propType = CONFIG >>>>>>>>>>> This is a logical switch

Note: This would not show why the host still have the portgroups, but it will confirm that the problem is in the host as it is advertising those.

Powered by Wolken Software