VTEP vmkernel ports are not deleted when moving ESXi from an NSX cluster to a non-NSX cluster

Products

• VMware NSX Networking

Issue/Introduction

The KB is to explain possible issue when moving ESXi from an NSX cluster to a non NSX cluster and how to mitigate the issue.

Symptoms:
Your NSX Manager is 6.4.2 or later.
When you move ESXi from an NSX cluster to a non-NSX cluster, you may experience following symptoms: 

• NSX VIBs are uninstalled successfully.
• VTEP vmkernel ports are not deleted.
• In the vsm.log file of NSX Manager, you see entries similar to:

2019-09-03 06:15:35.883 GMT  INFO DCNPool-8 VimNotificationHandler:106 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Received DELETE notification for object HOST_SYSTEM/host-2065

2019-09-03 06:15:35.938 GMT  INFO DCNPool-8 NwfabricInventoryDataChangeHandler:108 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Host host-2065 deleted. Marking as unprepared.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX Data Center for vSphere 6.4.x

Cause

Since NSX 6.4.2, NSX Manager has updated how to fetch inventory updates from vCenter Server.
The new method is not deterministic, and NSX Manager might receive DELETE notification from vCenter Server instead of MODIFY.
If it is the case, NSX Manager can not clean up VTEP vmkernel ports.

Note:
It is a timing issue and not always seen.
It might be observed more frequently under certain conditions including vCenter Server version and operations.

Resolution

Currently, there is no resolution.

Workaround:
You can revert how NSX Manager gets inventory updates from vCenter Server as a workaround.
Contact VMware support for the workaround.
 

Note:
The change was implemented to avoid issues under a certain condition.
Faulty clients sometimes create tremendous number of ContainerView in vCenter Server and NSX Manager might have trouble in fetching inventory from vCenter Server.

If you revert it, you might see some issues in case such a faulty client is connected to the vCenter Server.

You can also manually delete VTEP vmkernel ports and release assigned IP addresses after you experience the symptom.

• VTEP vmkernel ports can be deleted through vSphere Web Client.
• IP addresses can be released by REST API. Refer to "Working With IP Pool Grouping Objects" in NSX API Guide for more details.

Additional Information

Impact/Risks:
Because VTEP vmkernel ports are not deleted, if you move the ESXi to an NSX cluster again, new VTEP vmkernel ports are created and thus the VTEP vmkernel ports are doubled.

If you have assigned IP addresses to the VTEP from IP address pool, the assigned IP addresses are not released to IP address pool.
As a result, the IP address pool might be exhausted.