Additional NSX-T manager nodes are stuck at "VM Clustering in Progress"
search cancel

Additional NSX-T manager nodes are stuck at "VM Clustering in Progress"

book

Article ID: 345857

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:
  • After attempting to add one or more manager nodes to an NSX-T 2.4 installation, the additional node never move past the "VM Clustering in Progress" state in the UI.
  • Refreshing the UI, rebooting one or more manager nodes or deleting one more manager nodes does not change the behavior in the UI.
  • You see that all manager nodes are UP when you run get cluster status from the command line on one of the manager nodes.


Environment

VMware NSX-T Data Center 2.x
VMware NSX-T Data Center

Resolution

To resolve this issue, delete the nodes that were added and remove their deployments.
  1. Log in to the original NSX-T manager node (console or SSH) as the admin user.
  2. Issue the following command to retrieve the UUID values for each of the manager nodes:
get cluster status

Note: You will see output similar to the following:

Cluster Id: axxxxxx2-5xx3-4xx6-bxx5-9xxxxxxxxxx2
Group Type: DATASTORE
Group Status: STABLE

Members:
    UUID                                       FQDN                                       IP               STATUS
    6xxxxxx2-3xx2-axx8-5xxc-2xxxxxxxxxxb       nsxmgr-01a                                 192.168.xx.42   UP
    5xxxxxxe-3xx8-4xxa-8xxd-1xxxxxxxxxx6       nsxmgr-02a                                 192.168.xx.43   UP
    3xxxxxx9-bxxd-4xxb-9xx1-2xxxxxx7       nsxmgr-03a                                 192.168.xx.44   UP

Group Type: CLUSTER_BOOT_MANAGER
Group Status: STABLE

Members:
    UUID                                       FQDN                                       IP               STATUS
    5xxxxxxe-3xx8-4xxa-8xxd-1xxxxxxxxxx6       nsxmgr-02a                                 192.168.xx.43   UP
    6xxxxxx2-3xx2-axx8-5xxc-2xxxxxxxxxxb       nsxmgr-01a                                 192.168.xx.42   UP
    3xxxxxx9-bxxd-4xxb-9xx1-2xxxxxx7       nsxmgr-03a                                    192.168.xx.44   UP

Group Type: CONTROLLER
Group Status: STABLE

Members:
    UUID                                       FQDN                                       IP               STATUS
    bxxxxxx4-cxxb-4xxb-axxf-5xxxxxxxxxx0       nsxmgr-02a                                 192.168.xx.43   UP
    9xxxxxxd-exx9-4xxc-bxx9-4xxxxxxxxxx0       nsxmgr-03a                                 192.168.xx.44   UP
    cxxxxxx2-3xx0-4xxb-bxxd-9xxxxxxxxxx2       nsxmgr-01a                                 192.168.xx.42   UP

Group Type: MANAGER
Group Status: STABLE

Members:
    UUID                                       FQDN                                       IP               STATUS
    5xxxxxxe-3xx8-4xxa-8xxd-1xxxxxxxxxx6       nsxmgr-02a                                 192.168.xx.43   UP
    6xxxxxx2-3xx2-axx8-5xxc-2xxxxxxxxxxb       nsxmgr-01a                                 192.168.xx.42   UP
    3xxxxxx9-bxxd-4xxb-9xx1-2xxxxxx7       nsxmgr-03a                                    192.168.xx.44   UP

Group Type: POLICY
Group Status: STABLE

Members:
    UUID                                       FQDN                                       IP               STATUS
    5xxxxxxe-3xx8-4xxa-8xxd-1xxxxxxxxxx6       nsxmgr-02a                                 192.168.xx.43   UP
    6xxxxxx2-3xx2-axx8-5xxc-2xxxxxxxxxxb       nsxmgr-01a                                 192.168.xx.42   UP
    3xxxxxx9-bxxd-4xxb-9xx1-2xxxxxx7       nsxmgr-03a                                    192.168.xx.44   UP

Group Type: HTTPS
Group Status: STABLE

Members:
    UUID                                       FQDN                                       IP               STATUS
    5xxxxxxe-3xx8-4xxa-8xxd-1xxxxxxxxxx6       nsxmgr-02a                                 192.168.xx.43   UP
    6xxxxxx2-3xx2-axx8-5xxc-2xxxxxxxxxxb       nsxmgr-01a                                 192.168.xx.42   UP
    3xxxxxx9-bxxd-4xxb-9xx1-2xxxxxx7           nsxmgr-03a                                    192.168.xx.44   UP

Note:  Make a note of the UUID values corresponding to any manager nodes that are stuck at "VM Clustering in Progress".
  1. Issue a command similar to the following to remove one of the problematic manager nodes.nodes:
detach node 3xxxxxx9-bxxd-4xxb-9xx1-2xxxxxx7

Note: Replace 3xxxxxx9-bxxd-4xxb-9xx1-2xxxxxx7  with the appropriate UUID value noted in Step 2.
  1. Repeat Step 3 if there is another node that is also stuck.
  2. From a system with the curl command installed and access to the original NSX-T manager VM, issue a command similar to the following to delete the deployment of one of the problematic manager nodes:
curl -k -u admin:VMware1! -X POST https://192.168.xx.42/api/v1/cluster/nodes/deployments/3xxxxxx9-bxxd-4xxb-9xx1-2xxxxxx7?action=delete&force_delete=true"
Note: Replace 192.168.xx.42 with the IP address of the original NSX-T manager VM and replace 3xxxxxx9-bxxd-4xxb-9xx1-2xxxxxx7  with the appropriate UUID value noted in Step 2.
  1. Repeat Step 5 if there is another node that is also stuck.


Powered by Wolken Software