NSX Edge Deployment fails on SDDC Manager when trying to retrieve NSX Inventory Data

Products

• VMware SDDC Manager

Issue/Introduction

Symptoms:

 

• During the workflow we are getting an error similar to "Unable to get NSX related data from inventory - Expected 1 but did not find any MORs of a type DistributedVirtualPortgroup and name xxx"
  
  
• SDDC manager contains errors similar to the excerpt below:
  
  /var/log/vmware/vcf/domainmanager/domainmanager.log
  ERROR [vcf_dm,757881f5d620466e,e653] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-15]  [1Q9B66] FAILED_TO_FETCH_NSXT_DATA_FROM_INVENTORY Unable to get NSX related data from inventory
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Unable to get NSX related data from inventory
..
Caused by: java.lang.IllegalArgumentException: Expected 1 but did not find any MORs of a type DistributedVirtualPortgroup and name VCF-edge_stale_nsx_portgroup-External-1
  
  

Environment

VMware Cloud foundation 5.x

Cause

• The issue occurs due to a PortGroup data mismatch between vCenter and SDDC manager inventory
  
  
• This is most likely to happen if a PortGroup has been created outside of SDDC manager.

Resolution

To resolve the issue remove the stale PortGroup from the VCF inventory using the steps below:

 

IMPORTANT NOTE: Make sure that backup/snapshots of the SDDC manager have been taken prior to any modification.

1. Confirm the port group retrieved from the logs DO NOT exist in vCenter.
   
   
   
2. Retrieve the VDS ID that the portgroups are on. This can be done using the following command:
   
    curl -s localhost/inventory/vds | jq
   
   Note the ID as you'll need it later.
   
   
   
3. Create a JSON update spec using the following command:
   
   

   curl -s localhost/inventory/vds?id=<ID> | jq ".[0]" > vds_info.json
   
   Example: 
   curl -s localhost/inventory/vds?id=7c764c6-7dd9-4b7c-b88e-3feab8f190f6 | jq ".[0]" > vds_info.json

   This should create a json with the VDS that we want to update under a file called vds_info.json.
   
   

4. Open up vds_info.json with a vi editor and remove the json entries with the name of the stale portgroup. 
   
   

   The information that we want to delete should be like the excerpt below. Please make sure there are no extra commas or brackets left over when removing these entries.

       {
            "name": "VCF-edge_stale_nsx_portgroup-External-1",
            "transportType": "EXTERNAL",
            "type": "EPHEMERAL",
            "vlanId": 0,
            "activeUplinks": [
               "uplink1"
            ],
            "standbyUplinks": [
               "uplink2"
            ]  
         }, 
   
   Once done save the file.
   
   

5. After performing step 5 verify the json is still a valid json string. You can use the following command to check:
   
    cat vds_info.json | jq

If there is no error, it should print the json out
   
   

6. Update the vds table in the Inventory using the updated vds_info.json file using the following command:
   
   curl localhost/inventory/vds/<id> -X PUT -d @vds_info.json -H 'Content-Type: application/json'
   
   Example:
   curl localhost/inventory/vds/7c764c6-7dd9-4b7c-b88e-3feab8f190f6 -X PUT -d @vds_info.json -H 'Content-Type: application/json'
   
   

7. Retry the creation workflow.