Converging existing infrastructure with NSX fails with 'NSXT_CLUSTER_NOT_FOUND'

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When attempting to converge existing infrastructure with NSX in VMware Cloud Foundation (VCF), the process fails during the task "Generate inputs for configuring nsx after Import."
• The SDDC Manager /var/log/vmware/vcf/domainmanager/domainmanager.log file contains the following error stack:

  YYYY-MM-DDTHH:MM:SS:12.604+0000 DEBUG [vcf_dm,######################,b8c1] [c.v.v.v.f.a.GeneratePostNsxImportConfigurationsInputAction,dm-exec-20]  NSX Address received: nsxvip.example.com
  YYYY-MM-DDTHH:MM:SS:12.604+0000 DEBUG [vcf_dm,######################,b8c1] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,dm-exec-20]  Fetching NSX Clusters data from inventory
  YYYY-MM-DDTHH:MM:SS:12.606+0000 DEBUG [vcf_dm,######################,b8c1] [c.v.v.v.f.a.GeneratePostNsxImportConfigurationsInputAction,dm-exec-20]  NSX Cluster List fetched: [{"id":"######-####-####-##########","domainIds
  ":["######-####-####-##########"],"status":"ACTIVE","version":"4.2.3.3.0-25171318","clusterFqdn":"##.##.##.##","nsxtClusterDetails":[{"id":"######-####-####-##########","vmName":"######-####-####-##########","fqdn":"##.##.##.##"},{"id":"######-####-####-##########","vmName":"######-####-####-##########","fqdn":"##.##.##.##"},{"id":"######-####-####-##########","vmName":"######-####-####-##########","fqdn":"##.##.##.##"}],"shared":true}]
  YYYY-MM-DDTHH:MM:SS:12.609+0000 ERROR [vcf_dm,######################,b8c1] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-20]  [IVI9DK] NSXT_CLUSTER_NOT_FOUND NSX cluster not found
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: NSX cluster not found

Environment

VCF 9.x

Cause

•  This issue is typically caused by a DNS reverse lookup failure for the NSX Virtual IP (VIP). If the SDDC Manager cannot resolve the FQDN associated with the NSX cluster IP in its inventory, the convergence task fails to identify the existing NSX cluster.

Resolution

•  To resolve this issue, manually update the SDDC Manager platform database to ensure the cluster_fqdn entry matches the correct NSX VIP FQDN.
  
  1. Take a snapshot of the SDDC Manager VM.
  2. SSH into the SDDC Manager with vcf user and then witch to root user.
  3. Connect to the Platform data-base:

  psql -h localhost -U postgres -d platform

  4. Retrieve  the NSX-T cluster details:

  select * from nsxt;

  Sample output:

  platform=# select * from nsxt;
  
  id                   | ########-####-####-####-############ ---------> Note this id (this will be used in below APIs)
  creation_time        | #############
  modification_time    | #############
  status               | ACTIVE
  version              | 9.2.0.0.25277109
  cluster_fqdn         | ##.##.##.##
  is_shared            | t
  nsxt_cluster_details | [{"fqdn":"##.##.##.##","vmName":".example-nsx-manager","id":"########-####-####-####-############"}]
   

  5. Exit the database.

  \q

  6. Fetch the current inventory JSON using the ID from Step 4

  curl -s -X GET "http://localhost/inventory/nsxt?id=<ID_FROM_STEP_4>" -H "Content-Type: application/json" | json_pp > updatednsxt.json

  7. Update the json payload and set fqdn to correct NSX node FQDN.

  vi updatednsxt.json

  Note: Post required modification been done, ensure to remove 'square bracket' from first and last line of updatednsxt.json file before proceeding to Step 8.
  
  8. Update the platform database using the modified JSON payload:

  curl -s -X PUT "http://localhost/inventory/nsxt/<ID_FROM_STEP_4>" -H "Content-Type: application/json" -d @updatednsxt.json
  
  

  9. Make sure only fields cluster_fqdn, fqdn (in nsxt_cluster_details) in 'nsxt' table are updated.

  psql -h localhost -U postgres -d platform

  select * from nsxt;

  
  11. Retry the convergence task from the VCF installer UI.