NSX Edge Deployment fails on SDDC Manager when trying to retrieve NSX Inventory Data

Products

• VMware SDDC Manager

Issue/Introduction

• During the workflow we are getting an error similar to "Unable to get NSX related data from inventory - Expected 1 but did not find any MORs of a type DistributedVirtualPortgroup and name xxx"
  
  
• SDDC manager contains errors similar to the excerpt below:
  
  /var/log/vmware/vcf/domainmanager/domainmanager.log
  

  ERROR [vcf_dm,#############,e653] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-15]  [1Q9B66] FAILED_TO_FETCH_NSXT_DATA_FROM_INVENTORY Unable to get NSX related data from inventory
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Unable to get NSX related data from inventory
  Caused by: java.lang.IllegalArgumentException: Expected 1 but did not find any MORs of a type DistributedVirtualPortgroup and name VCF-edge_stale_nsx_portgroup-External-1

  
  
  

Environment

VMware Cloud Foundation 5.x

Cause

• The issue occurs due to a PortGroup data mismatch between vCenter and SDDC manager inventory
  
  
• This is most likely to happen if a PortGroup has been created outside of SDDC manager.

Resolution

To resolve the issue remove the stale PortGroup from the VCF inventory using the steps below:

IMPORTANT NOTE: Make sure that backup/snapshots of the SDDC manager have been taken prior to any modification.

1. Confirm the port group retrieved from the logs DO NOT exist in vCenter.
   
   
2. Retrieve the VDS ID that the portgroups are on. This can be done using the following command:
   

   curl -s localhost/inventory/vds | jq

   
   Note the ID as you'll need it later.
   
   
   
3. Create a JSON update spec using the following command:
   
   

   curl -s localhost/inventory/vds?id=<ID> | jq ".[0]" > vds_info.json

   Example: 

   curl -s localhost/inventory/vds?id=######-####-####-####-##########| jq ".[0]" > vds_info.json

   
   This should create a json with the VDS that we want to update under a file called vds_info.json.
   
   

4. Open up vds_info.json with a vi editor (vi vds_info.json) and remove the json entries with the name of the stale portgroup(To edit the file, press ESC, then press i.). 
   
   

   The information that we want to delete should be like the excerpt below. Please make sure there are no extra commas or brackets left over when removing these entries.

    {
               "name": "VCF-edge_stale_nsx_portgroup-External-1",
               "transportType": "EXTERNAL",
               "type": "EPHEMERAL",
               "vlanId": 0,
               "activeUplinks": [
                  "uplink1"
               ],
               "standbyUplinks": [
                  "uplink2"
               ]  
     }, 
    

   Once done save the file (ESC + :wq!)
   
   

5. After performing step 5 verify the json is still a valid json string. You can use the following command to check:
   
    

   cat vds_info.json | jq

   

If there is no error, it should print the json output
   
   

6. Update the vds table in the Inventory using the updated vds_info.json file using the following command:
   
   

   curl localhost/inventory/vds/<id> -X PUT -d @vds_info.json -H 'Content-Type: application/json'
   Example:
   curl localhost/inventory/vds/######-####-####-######### -X PUT -d @vds_info.json -H 'Content-Type: application/json'

    

7. Retry the creation workflow.

Additional Information

In case it still fails on the next workflow with below error:

VCF_ERRORS_GENERIC_INPUT_PARAM_ERROR Invalid parameter: {0}

Use the script to cleanup VMware Cloud Foundation NSX-T Edge Cluster Deployment Removal Tool