VMs Disappear from NSX Inventory after migrating to sepecific ESXi hosts in the cluster.
search cancel

VMs Disappear from NSX Inventory after migrating to sepecific ESXi hosts in the cluster.

book

Article ID: 434380

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • VMs initially lost configured NSX tags after which they disappeared from the NSX inventory when they were migrated to specific ESXi hosts.
  • The VMs would come back under NSX, once they are migrated off these ESXi hosts.
  • All the problematic ESXi hosts were reported to be managed by the same NSX manager node. 
  • It was validated by running the command: get transport-nodes status

<Date>
TransportNode-ID                       Remote-Address                     Controller                             Manager-IP     Manager-IPv6               SSL-Enabled  Connection-State  Supported-Versions   Node-Type    Name
<Node_UUID>                            <Host Remote Address>              <controller_uuid>                      <Manager2_IP>  <Manager2_IPv6>            true         OPENED            [4.1, 4.0, 4.2, 3.2] ESXi         <Host-name>
<Node_UUID>                            <Host Remote Address>              <controller_uuid>                      <Manager2_IP>  <Manager2_IPv6>            true         OPENED            [4.1, 4.0, 4.2, 3.2] ESXi         <Host-name>

  • In the NSX UI, the concerned manager node reported a degraded cluster status with the cm-inventory service of the node as down:

Group Type: CM-INVENTORY
Group Status: DEGRADED

Members:
    UUID                            IPV4             IPV6                                      STATUS           FQDN
    <UUID>                          <Manager1_IP>    <Manager1_IPv6>                           UP               <Manager1_FQDN>
    <UUID>                          <Manager2_IP>    <Manager2_IPv6>                           DOWN             <Manager2_FQDN>
    <UUID>                          <Manager3_IP>    <Manager3_IPv6>                           UP               <Manager3_FQDN>

  • The NSX Manager syslog reported an exception when trying to fetch the vCenter inventory details:

YYYY-MM-DDTHH:MM:SS <Host> NSX 6016 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="manager"] UserName: <user>' ModuleName:'Policy' Operation:'GET@/api/v1/discovered-resources/enforcement-points/default/compute-managers' Operation status: 'failure' Error: Error occurred while executing request on enforcement point path=[/infra/sites/default/enforcement-points/default]: com.vmware.nsx.management.inventory.restclient.InventoryRestException: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:7443/cm-inventory/api/v1/fabric/compute-managers": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused).

  • The cm-inventory tomcat-wrapper logs reported the service has crashed due to memory exhaustion:

INFO   | jvm 1    | 2025/06/26 17:28:16 | # java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper  | 2025/06/26 17:28:16 | The JVM has run out of memory.  Requesting thread dump.
....
STATUS | wrapper  | 2025/06/26 17:28:45 | Launching a JVM...
INFO   | jvm 2    | 2025/06/26 17:28:45 | WrapperManager: Initializing...
INFO   | jvm 2    | 2025/06/26 17:28:47 | java.lang.Error: org.apache.catalina.LifecycleException: Failed to initialize connector [Connector[HTTP/1.1-7443]]
INFO   | jvm 2    | 2025/06/26 17:28:47 |       at org.apache.catalina.startup.Catalina.load(Catalina.java:649)
INFO   | jvm 2    | 2025/06/26 17:28:47 |       at org.apache.catalina.startup.Catalina.load(Catalina.java:669)
INFO   | jvm 2    | 2025/06/26 17:28:47 |       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 2    | 2025/06/26 17:28:47 |       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 2    | 2025/06/26 17:28:47 |       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Environment

VMware NSX

Cause

  • The cm-inventory service on the NSX Manager experiences a Java Virtual Machine (JVM) OutOfMemoryError due to prolonged uptime of the manager node, causing the service to crash.
  • Consequently, the manager fails to retrieve VM data from the associated ESXi hosts resulting in VMs not being reported on NSX inventory.
  • When VMs remain inaccessible to the inventory service for more than 30 minutes, they are treated as new objects upon reconnection, resulting in the loss of their existing NSX tags.

Resolution

  • Reboot the affected NSX Manager node to clear the space usage, resulting in the restart of the required cm-inventory services.
  • Once all the services are back up on the manager node cluster, manually reapply the missing NSX-T tags to the affected VMs, or restore the tags by performing a restoration from a recent NSX-T Manager backup.
  • To validate NSX manager cluster health, run the command:
    • get cluster status

Additional Information

For more information regarding VMs losing NSX tags, please refer: NSX lost all the tags of VMs

Powered by Wolken Software