VCF Upgrade Precheck Fails on NSX Manager Phase Due to Resource Validation Timeout

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When upgrading VMware Cloud Foundation (VCF) (e.g., from version 9.1 to 9.1.1), the Lifecycle Manager (LCM) precheck fails during the NSX Manager planning phase.
• The VCF Ops UI displays a "Resource validation timeout" error.
  
  
  
  
• Reviewing the /var/log/vmware/vcf/lcm/lcm-debug.log in SDDC Manager reveals the following error:

  com.vmware.vcf.lcm.precheckagents.nsx.exception.NsxClusterRetrievalException: Failed to retrieve NSX-T manager for resourceId: [NSXT_MANAGER_FQDN]

Environment

• VMware Cloud Foundation 9.1.x
• VMware SDDC Manager

Cause

The NSX-T manager status in the SDDC Manager platform database is incorrectly set to "ACTIVATING" instead of "ACTIVE". The LCM precheck framework queries the platform database to establish a connection with the NSX-T cluster; because the status is not "ACTIVE," the framework fails to retrieve the cluster connection, leading to the validation timeout.

Resolution

To resolve this issue, update the NSX-T manager status in the platform database to "ACTIVE" by following these steps:

 

1. Take a snapshot of SDDC Manager VM
2. SSH into the SDDC Manager with vcf user and su to root
3. Get the NSXT cluster details

   curl -v -k http://127.0.0.1:7100/inventory/nsxt | json_pp

   Note the id of nsxt cluster
   
   Sample output:

   "domainIds" : [
            "ff38####-####-####-####-########24a4"
         ],
         "id" : "7611####-####-####-####-########c747", <=========== [NSXT Cluster ID]
         "nsxtClusterDetails" : [
            {
               "fqdn" : "nsxmgr01.example.com",
               "id" : "c569####-####-####-####-########8d9b",
               "vmName" : "nsxmgr01"
            }
         ],
         "shared" : true,
         "status" : "ACTIVATING",
         "version" : "9.1.0.0.25318225"
      }

    

4. Get the NSXT cluster details in a file

   curl -v -k http://127.0.0.1:7100/inventory/nsxt?=<NSXT Manager Cluster ID from Step#3> | json_pp > nsxt_manager.json

5. Edit the file > Remove the square brackets > Update status to "ACTIVE'

   vi nsxt_manager.json > Type 'i' for insert mode > Remove the square brackets from start and end of the file > Update status from "ACTIVATING" to "ACTIVE"

   
   Note: Do not make any other changes on the json file
   
   
6. Update the NSXT Manager status to ACTIVE

   curl -X PUT 'http://127.0.0.1:7100/inventory/nsxt/<NSXT Manager ID from Step#>' -H 'Content-Type:application/json' -d @nsxt_manager.json

7. Repeat step # 3 and confirm NSXT Manager status shows as 'ACTIVE'
8. Retry the precheck in SDDC Manager