Migrated VTEPs deleted when uninstalling NSX-V agency after migration

Products

• VMware NSX-T Data Center

Issue/Introduction

• NSX-T TEPs deleted unexpectedly after deleting the NSX-v EAM agency

         Relevant log’s location:
         EAM and VPXD logs in VC should have logs like below:

• /var/log/vmware/eam/eam.log:
  2024-07-12T16:26:02.897Z |  INFO | host-agent-3 | AgencyBase.java | 1321 | Destroying agency owned by Administrator vsphere.local (VSPHERE.LOCAL\Administrator) null::Agency:<UUID>
2024-07-12T16:26:02.898Z |  INFO | host-agent-3 | ComputeResourceBase.java | 323 | Unregistered listener: HostVMAgency(ID:'Agency:<UUID>:null') to [<UUID>::ClusterComputeResource:<cluster moid>]
2024-07-12T16:26:02.898Z |  INFO | host-agent-3 | VmCounter.java | 78 | Removed agency: HostVMAgency(ID:'Agency:<UUID>:null') from [<UUID>::ClusterComputeResource:<cluster moid>]

 

• /var/log/vmware/vpxd/vpxd.log:
  2024-07-12T16:26:04.755Z info vpxd[07449] [Originator@6876 sub=vpxLro opID=opId-13b71-2871912-4f] [VpxLRO] -- BEGIN task-1195129 -- networkSystem-19 -- vim.host.NetworkSystem.removeVirtualNic -- <UUID>)
2024-07-12T16:26:04.893Z info vpxd[07449] [Originator@6876 sub=vpxLro opID=opId-13b71-2871912-4f] [VpxLRO] -- FINISH task-1195129

Environment

NSX-T 3.X
NSX    4.X

Cause

NSX-V MP has a table xvs_vmknic_info in DB secureall to record the VTEPs of all NSX-V hosts.

When a NSX-V host is migrated to NSX-T, the V2T migrator issues a request "/api/2.0/vdn/config/host/<Host_ID>/vxlan/vteps?action=resync" to NSX-V MP at the last step of the host migration

NSX-V MP should delete the VTEPs of the host from this xvs_vmknic_info table because the VTEPs have been migrated to NSX-T.

If for some reason the VTEPs of the host were not deleted from the xvs_vmknic_info table, later when the ESX Agent Manager agency of the host's cluster is removed, a task will start in VC to delete the VTEPs in the host and overlay traffic to/from the VMs in the host will stop.

Resolution

This is a known issue impacting NSX

Workaround:

Go to NSXT UI "System" -> "Fabric" to find all host TNs whose VTEPs were deleted, edit each TN's config but do not change anything, save the TN config and wait for its state becomes success again. The VTEPs will be recreated.

Preventitive Steps:

After migration completes, clear stale VTEPs in NSXv MP DB:
    1. ssh into NSXv MP as root
    2. run below command to clear the DB table
       psql -U secureall -d secureall -c "delete from xvs_vmknic_info;"
    3. run below command to confirm the output shows 0 row
    psql -U secureall -d secureall -c "select * from xvs_vmknic_info;"
    4.EAM agency can safely be deleted.