NSX-T lost all the tags of VMs

Products

• VMware NSX

Issue/Introduction

VMs lose tags once they are unregistered for more than 30 minutes or VM is showing as inaccessible for longer than 30 minutes. If the VM is brought back into the inventory after 30 minutes, it would be treated as a new object, similarly if a vm is deleted from vCenter inventory and re-added even if this is done in under 30 minutes it will be a new object and lose its NSX tags.

Relevant logs to look for:
For VMware NSX-T Data Center 3.1.x and below, look in: /var/log/proton/nsxapi.log
For VMware NSX-T Data Center 3.2.x and above look in: /var/log/cm-inventory/cm-inventory.log

20XX-05-21T19:12:28.279Z INFO inventoryTasksScheduler-2 CleanupHandler - FABRIC [nsx@6876 comp="manager_host_name" level="INFO" subcomp="manager"] Successfully deleted data for DeletedObjectId 70####18-8###-###c-###b-b########8p0 of type VM
20XX-05-21T19:12:30.568Z INFO inventoryTasksScheduler-2 CleanupHandler - FABRIC [nsx@6876 comp="manager_host_name" level="INFO" subcomp="manager"] Successfully deleted data for DeletedObjectId 70####18-8###-###c-###b-b########8p0 2 of type VM
20XX-05-21T19:14:28.309Z INFO inventoryTasksScheduler-2 CleanupHandler - FABRIC [nsx@6876 comp="manager_host_name" level="INFO" subcomp="manager"] Successfully deleted data for DeletedObjectId 70####18-8###-###c-###b-b########8p0 of type VM

The following log line indicates that the 30 min timer is triggered for VM with external id 50####18-8###-###c-###b-b########8a0:

-----------------------
20XX-07-03T08:15:02.300Z INFO inventoryTasksScheduler1 VirtualMachineServiceImpl 4257 FABRIC [nsx@6876 comp="manager_host_name" level="INFO" subcomp="cm-inventory"] Marked VMContainer20####38-8###-###c-###b-b########2q0 as deleted with timestamp 16######02300
-----------------------

The following log line indicates that the 30 min time limit has been reached and the VM with external id 50####18-8###-###c-###b-b########8a0 is deleted:

-----------------------
20XX-07-03T08:46:03.837Z INFO inventoryTasksScheduler10 CleanupHandler 4257 SYSTEM [nsx@6876 comp="manager_host_name" level="INFO" subcomp="cm-inventory"] Successfully deleted data for DeletedObjectId 70####18-8###-###c-###b-b########8p0 of type VM
-----------------------

Environment

VMware NSX Data Center
VMware NSX

Cause

The Host has not reported the instance UUID for specified VMs in more than 30 minutes. The VM and associated tags are then removed from the NSX inventory.

This is the expected behaviour. If a VM is marked for deletion from a host and if that VM is not claimed by any other host within the 30 minutes timeframe then the VM will be removed from the NSX inventory and tags associated with that VM would also be removed from the VM.

Supporting documentation

Example scenario:  

• Prior to an outage, the host syncs with NSX and vCenter reporting 10 VMs.
• After an outage of multiple hours, when the host is back online and is able to sync with NSX, if it has 9 VMs, then it will send a full sync to NSX. As the host only has 9 VMs compared to the previous update of 10 VMs, NSX will mark the missing VM for deletion; it will wait for 30 minutes to see if any host claims  the mark for deleted VM, if no other host claims that VM in next 30 minutes then NSX will remove it from the inventory.
• If the VM is registered on a host after 30 minutes, it is treated as a new VM, and tags must be reapplied.

Resolution

This is the expected product behaviour.
In the scenario of a Virtual Machine losing its NSX tag(s) there are two workaround options:

• The NSX-T tags need to be manually applied to the affected VMs to ensure proper tagging and alignment with the network and security policies.
• Tags can also be restored by restoring from a NSX-T Manager backup.

Additional Information

VMs may lost communication due to DFW rules not being applied after losing NSX tags.