NSX-T tags assigned to VMs on NSX-T segment disappear after a VM is disconnected for more than 30 minutes

Products

• VMware NSX

Issue/Introduction

• After recovering from an outage or other event causing loss of connectivity between hosts, NSX-T, and vCenter you may notice tags previously applied to VMs in NSX-T are missing
• This behavior is expected and as per design. When a VM disappears from the NSX inventory for more than 30 minutes, NSX tags on the VM are lost. If the same VM reappears in the NSX inventory after 30 minutes, NSX treats it as a new VM, and you must add the tags again on the VM. This is referenced in the documentation for Tags.  
• Check /var/log/proton/nsxapi.log in VMware NSX 3.1.x and below, for VMware NSX 3.2.x and above check /var/log/cm-inventory/cm-inventory.log to see the series of events. Example below:

1. Delete event request from host <host-TN-ID> to delete VM <virtual machine ID>.

2023-05-03T10:07:12.913Z INFO task-executor-10 MessageBatchProcessorImpl 6211 FABRIC [nsx@6876 comp="nsx-manager" 
level="INFO" subcomp="manager"] Process msg from host <host-TN-ID> : CT_DELETE  
 esx_entry { 
 key: EIK_INSTANCE_UUID 
 value: "<virtual machine ID>"
 ...

2. Marked VM for deletion: 

2023-05-03T10:07:55.092Z INFO inventoryTasksScheduler-1 VirtualMachineServiceImpl 6170 FABRIC [nsx@6876 comp="nsx- 
manager" level="INFO" subcomp="manager"] Marking VMContainer <virtual machine ID> as deleted, it will be 
cleaned up late

3. No host claimed that VM in the next 30 minutes and the VM was deleted from NSX-T inventory and this is when the tags assigned to the VM were lost:

2023-05-03T10:39:01.078Z INFO inventoryTasksScheduler-1 VmCleanupHandler 6170 FABRIC [nsx@6876 comp="nsx-manager" 
level="INFO" subcomp="manager"] Deleting VM with id: <virtual machine ID>, deletion timestamp: 
1683108475092

4. Only after multiple hours did another host 6569a259-0516-433f-a685-2f3a2ad412d8  reported/claimed this VM: 

2023-05-03T14:29:56.769Z INFO task-executor-0 MessageBatchProcessorImpl 6170 FABRIC [nsx@6876 comp="nsx-manager" 
level="INFO" subcomp="manager"] Process msg from host <host-TN-ID> : CT_CREATE  
 esx_entry { 
 key: EIK_INSTANCE_UUID 
 value: "<virtual machine ID>" 
...

Environment

VMware NSX 3.x
VMware NSX

Cause

This behavior is expected and as per design. When a VM disappears from the NSX inventory for more than 30 minutes, NSX tags on the VM are lost. If the same VM reappears in the the NSX inventory after 30 minutes, NSX treats it as a new VM, and you must add the tags again on the VM. This is referenced in the documentation for Tags.  

 

An example of expected behavior:

• Before the outage, the host sync with NSX & vCenter reported 10 VMs.
• After an outage of multiple hours, when the host comes up & syncs up with NSXT if it has 9 VMs, then it will send a full sync to NSX. As it has only 9 VMs compared to the previous update of 10 VMs, NSXT will mark the missing VM for deletion; it will wait for 30 min to see if any host claims the marked-for-deletion VM, if no other host claims that VM within 30 mins, then NSX will remove it from the inventory.
• If the VM is registered on a host after 30 minutes, it is treated as a new VM, and tags must be reapplied.

Resolution

• The NSX-T tags need to be manually applied to the affected VMs to ensure proper tagging and alignment with the network and security policies.
• Tags can also be restored by restoring from a NSX-T Manager backup. 

Additional Information