Duplicate VMs present in NSX inventory
search cancel

Duplicate VMs present in NSX inventory

book

Article ID: 413910

calendar_today

Updated On:

Products

VMware NSX VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

  • The SDDC Manager was upgraded to 4.5.2. The next step was to apply the configuration drift bundle for VMware Cloud Foundation 4.5.2.0.
    This activity failed within one to two minutes of being started, with the following UI error being shown (#### corresponds to the management domain name) :
    #### domain failed upgrade at Mmm dd, yyyy, hh:mm:ss. Please resolve the above upgrade failure for this bundle before applying any other available bundle

  • The following is observed in the SDDC Manager migration app upgrade log:
    /var/log/vmware/vcf/lcm/thirdparty/upgrades/####-####-8b37c85beb2f/sddcmanager-migration-app/logs/sddcmanager_migration_app_upgrade_yyyy-mm-dd_hh-mm-ss.log

     
    <timestamps>.370+0000 INFO  [vcf_migration,0000000000000000,0000] [c.v.v.c.f.p.n.h.EdgeNodeHostIdService,pool-5-thread-9]  Getting transport node status for TN id:  ###-####-dcdc12a2b526
<timestamps>.416+0000 INFO  [vcf_migration,0000000000000000,0000] [c.v.v.c.f.p.n.h.EdgeNodeHostIdService,pool-5-thread-9]  getEdgeNodeVmHostIdFromAnyNsxVersion: vm_uuid = ###-####-99200288ce97
<timestamps>.444+0000 INFO  [vcf_migration,0000000000000000,0000] [c.v.v.c.f.p.n.h.EdgeNodeHostIdService,pool-5-thread-9]  getEdgeNodeVmHostId: vmResultCount = 2
<timestamps>.444+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.v.c.n.s.c.c.ApiConnection,pool-5-thread-9]  Closed ApiClient connection.
<timestamps>.445+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,pool-5-thread-9]  [6CCHJ1] EDGE_NODE_HOST_INFO_COLLECTION_FAILED Failed to collect edge cluster node host info maps
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to collect edge cluster node host info maps
Caused by: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Edge node <edge-vm-name> lookup by ID ###-####-99200288ce97 returned 2 results, expected 1

  • NSX UI > System > Fabric > Nodes > Edge Transport Nodes shows correct Edge information.

  • NSX UI  >> Inventory > Virtual Machines  shows stale/duplicate entries for edge VM's and also other VM's .

  • In NSX Manager, you would notice similar logs in cm-inventory [/var/log/cm-inventory/cm-inventory.log] :

    <timestamps>  WARN task-executor-8 MessageBatchProcessorImpl 4169 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="cm-inventory"] Issue while processing VM update for VM ID [####-####-309e5379bcc0] - MissingVmInventoryException
<timestamps>  WARN task-executor-8 InventoryDiscoveryServiceImpl 4169 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="cm-inventory"] Caught Exception: com.vmware.nsx.management.cminv.common.exceptions.InventoryException: Issues during processing of 2 messages from host ####-####-cc5328fad9ba , skipped these messages.. Rethrowing.
com.vmware.nsx.management.cminv.common.exceptions.MissingVmInventoryException: Unable to find virtual machine with external identifier ####-####-309e5379bcc0.

     

Environment

VMware NSX 4.1.1 or later
VMware Cloud Foundation 4.5.2
SDDC Manager
NSX-T 3.2.3 or later

Cause

  • Upgrade failed due to duplicate edge VM entry in NSX inventory.
  • This issue occurred due to a race condition. For a subset of VMs, VmPropertiesMsg was present but VmContainerMsg was missing. This led to MissingInventoryException during processing of full sync message.

Resolution

This issue is resolved in NSX-T 3.2.4 and later, available at Broadcom downloads.
This issue is resolved in VMware NSX 4.1.2 & 4.2.0 and later, available at Broadcom downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

WORKAROUND:
If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.

 

Additional Information

Powered by Wolken Software