NSX-T tags assigned to VMs on NSX-T segment disappear after a VM is disconnected for more than 30 minutes

Products

• VMware NSX Networking

Issue/Introduction

• After recovering from an outage or other event causing loss of connectivity between hosts, NSX-T and vCenter you may notice tags previously applied to VMs in NSX-T are missing
• This behavior is expected and as per design. When a VM disappears from the NSX inventory for more than 30 minutes, NSX tags on the VM are lost. If the same VM reappears in the the NSX inventory after 30 minutes, NSX treats it as a new VM, and you must add the tags again on the VM. This is referenced in the documentation for Tags.  
• Check /var/log/proton/nsxapi.log in VMware NSX 3.1.x and below, for VMware NSX 3.2.x and above check /var/log/cm-inventory/cm-inventory.log to see the series of events. Example below:

1. Delete event request from host 58afd56f-9b78-4e73-bd92-00d0d51ed370 to delete VM 5028cd45-d7bd-6a1b-373d-fe2d67a41e89.

2023-05-03T10:07:12.913Z INFO task-executor-10 MessageBatchProcessorImpl 6211 FABRIC [nsx@6876 comp="nsx-manager" 
level="INFO" subcomp="manager"] Process msg from host 58afd56f-9b78-4e73-bd92-00d0d51ed370 : CT_DELETE  
 esx_entry { 
 key: EIK_INSTANCE_UUID 
 value: "5028cd45-d7bd-6a1b-373d-fe2d67a41e89"
 ...

2. Marked VM for deletion: 

2023-05-03T10:07:55.092Z INFO inventoryTasksScheduler-1 VirtualMachineServiceImpl 6170 FABRIC [nsx@6876 comp="nsx- 
manager" level="INFO" subcomp="manager"] Marking VMContainer 5028cd45-d7bd-6a1b-373d-fe2d67a41e89 as deleted, it will be 
cleaned up late

3. No host claimed that VM in next 30 mins and the VM was deleted from NSX-T inventory and this is when the tags assigned to the VM were lost:

2023-05-03T10:39:01.078Z INFO inventoryTasksScheduler-1 VmCleanupHandler 6170 FABRIC [nsx@6876 comp="nsx-manager" 
level="INFO" subcomp="manager"] Deleting VM with id: 5028cd45-d7bd-6a1b-373d-fe2d67a41e89, deletion timestamp: 
1683108475092

4. Only after multiple hours another host 6569a259-0516-433f-a685-2f3a2ad412d8  reported/claimed this VM: 

2023-05-03T14:29:56.769Z INFO task-executor-0 MessageBatchProcessorImpl 6170 FABRIC [nsx@6876 comp="nsx-manager" 
level="INFO" subcomp="manager"] Process msg from host 6569a259-0516-433f-a685-2f3a2ad412d8 : CT_CREATE  
 esx_entry { 
 key: EIK_INSTANCE_UUID 
 value: "5028cd45-d7bd-6a1b-373d-fe2d67a41e89" 
...

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center
VMware NSX-T Data Center 4.x

Resolution

• The NSX-T tags need to be manually applied to the affected VMs to ensure proper tagging and alignment with the network and security policies.
• Tags can also be restored by restoring from a NSX-T Manager backup. 

Additional Information

NSX-T Tags lost after VMs are lost from inventory for longer than 30 minutes/VMs with NSX-T Tags, lost communication after the vCenter was shutdown.