'Failed' status for VCF NSX 9.0.x upgrade Precheck
search cancel

'Failed' status for VCF NSX 9.0.x upgrade Precheck

book

Article ID: 433917

calendar_today

Updated On:

Products

VMware NSX VMware SDDC Manager / VCF Installer VMware vSphere ESXi

Issue/Introduction

  • VCF 9.0 upgrade precheck fails in NSX with no description, just a Failed status.

  • In the Fleet Management, Lifecycle page for the domain, where the precheck is being run, there is a message saying:

<NSX-Manager-FQDN> - NSX manager has a common name that does not match its hostname. Ensure that NSX manager has a valid certificate with its common name matching the host name.

  • In VCF Fleet Operations, Certificate management, one of the NSX managers is not displayed.
  • The domain was deployed with three NSX managers.
  • One of the NSX managers failed and was replaced in NSX, not SDDC manager.

Environment

VMware VCF 9.0.x

Cause

In VCF 9.0.x, redeploying an NSX manager in NSX is out of band and not supported.

Resolution

Ensure there is a backup of the NSX manager cluster.

SDDC does not sync back from NSX in current versions, in order to get the 3 node cluster back, we now need to reduce the cluster to a single node cluster, then scale out the cluster again in SDDC manager.

Ensure the repository is working and on the remaining node.

  1. SSH as admin user to one of the remaining NSX managers and run: get service install-upgrade
    • This will return an IP address of the manager the service is running on.
  2. If not the current node, ssh as admin user to the node that was displayed in the result.
  3. If the node displayed in the result, is the failed node, ssh as admin user to one of the good nodes and run: set repository-ip
  4. Confirm the service is now running on that node: get service install-upgrade
  5. Now we need to get the UUID of each manager node, as admin user run: get cluster config
  6. This will list each node in the cluster and its UUID, now detach the nodes one by one using the command: detach node <UUID>
  7. Once the command completes, run get cluster config and get cluster status and confirm the node is released and the now 2 node cluster is healthy.
  8. Repeat the process for the next node: detach node <UUID> 
  9. Once the command completes, run get cluster config and get cluster status and confirm the node is released and the now 1 node cluster is healthy.

Note: Detailed steps on for detaching a manager node can be found here Resize an NSX Manager Node

 

Next on the SDDC manager, we need to reduce the cluster to one node and then scale it out to three nodes.

To get the SDDC manager UUID for the NSX cluster:

    1. Log in to the SDDC Manager UI.
    2. Navigate to the Developer Center via the home dashboard or sidebar.
    3. Select the API Categories
      1. NSX-T Cluster

GET /v1/nsxt-clusters

        • Copy the UUID of the NSX-T that needs to scale out.

  • Next we need to use the following API to tell the SDDC manager, there is now only one NSX manager node, use the details from the single remaining NSX manager node.

PUT http://<sddc-manager-ip>/inventory/nsxt/########-2373-4c74-ba99-############

With the following data:

{
"id": "########-2373-4c74-ba99-############", -> This is the NSX-T cluster domain ID as seen in SDDC from the GET API above.
"domainIds": [
"########-4109-47b2-ae1c-############" -> This is the SDDC manager domain ID.
],
"status": "ACTIVE",
"version": "9.0.1.0.24952111", -> Replace with correct NSX-T version.
"clusterFqdn": "<NSX-VIP-FQDN>", -> Replace with NSX cluster VIP FQDN address.
"nsxtClusterDetails": [
{
"fqdn": "NSX-Manager-FQDN>", -> Replace with the FQDN of the remaining NSX manager.
"vmName": "<NSX-Manager-VMName", -> Replace with NSX manager VM name.
"id": "########-8bd0-01df-24b0-############" -> UUID of the remaining manager.
}
],
"shared": false -> Update based on SDDC manager GET cluster response above.
}

  • Now we can scale out the NSX cluster in the SDDC manager to three NSX manager nodes:
    1. Locate the NSX-T Clusters section and select scale-out endpoint as below:

      1. NSX-T Cluster

      2. POST /v1/nsxt-clusters/{nsxt-cluster-id}/scale-out

        1. Paste the nsxt-cluster-id

        2. *EXAMPLE* In the request body field, enter YOUR NODE DETAILS the following JSON payload:
          {
            "nsxManagerSpecs": [
              {
                "name": "<NAME>",
                "networkDetailsSpec": {
                  "ipAddress": "192.168.1.###",
                  "dnsName": "<FQDN>",
                  "gateway": "192.168.1.###",
                  "subnetMask": "255.255.255.0"
                }
              },
              {
                "name": "<NAME>",
                "networkDetailsSpec": {
                  "ipAddress": "192.168.1.###",
                  "dnsName": "<FQDN>",
                  "gateway": "192.168.1.###",
                  "subnetMask": "255.255.255.0"
                }
              }
            ]
          }

Click Execute

Note: Above, replace FQDN, NAME and IP addressing with the correct values for the two additional NSX manager node to add to the NSX manager cluster.

  • Post-validation (Full Synchronization may take up to an hour.)
    1. SDDC Manager UI
      1. NSX shows as healthy
        • Inventory now reflects 3 NSX Managers
      2. NSX UI
        • System > Appliances: 3 managers, stable cluster