NSX-T Multi NSX setup vSphere cluster is locked
search cancel

NSX-T Multi NSX setup vSphere cluster is locked

book

Article ID: 322461

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:

  • This is a Multi NSX setup, that is you have multiple NSX-T managers managing the same compute manager (vCenter).
  • The version of NSX-T is 3.2.2 or higher.
  • The vSphere cluster or host shows as locked in NSX-T 'System - Fabric - Nodes' even if it is not prepared in any NSX-T manager.
  • This may happen if a prepared cluster is unprepared and fails to release NSX-T ownership of the cluster by the 1st NSX-T manager. In the UI of the 2nd NSX-T manager you may see the below error message:
    This cluster is owned by another instance of NSX and can be viewed in read-only mode.
    Configuration data that is inaccessible and cannot be retrieved is marked Unknown.
  • In the Clusters view, the cluster has a lock symbol showing on it:

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware NSX-T Data Center

Cause

This issue occurs due to a race condition.
When the last TN (Transport node) is removed from the cluster (that is the last host is unprepared for NSX-T), the ownership should be released automatically.
In certain instances the ownership is not released.
This results in the cluster remaining locked by that NSX-T manager and prevents the cluster from being prepared by the other NSX-T manager.

Resolution

This is a known issue impacting NSX-T data center.

Workaround:
If the NSX-T manager which originally had ownership of the cluster is still present, proceed with step 1 here.
If the NSX-T manager which originally had ownership of the cluster is no longer present, due to failure or decommission, proceed to step 2.
    
Step 1
If the previous NSX-T manager is still accessible and not able to release the cluster ownership, make sure there are no TN's in the UI, that is there are no hosts prepared for NSX-T.

a. You can use the following API if the NSX-T Manager UI shows 'Not Configured' for the hosts present in the locked cluster.

  • To get the state of all TNs present in that manager
GET https://<NSX-Manager>/api/v1/transport-nodes/state
  • Check if any of the TN's are in a problematic state.

b. If any of the TN's are still present and stuck during uninstallation, remove the TN using the following force API:

DELETE https://<NSX-Manager>/api/v1/transport-nodes/6665####-7b5c-####-9512-9ac6####64fe?unprepare_host=false&force=true
  • Once this API completes, check if TN has been removed completely using the following API
GET https://<NSX-Manager>/api/v1/transport-nodes/<tn_id>/state

Repeat step 1.b for any TN not released.
If ownership is still not released, no hosts are prepared and the cluster is locked, proceed to step 2.

Step 2
There are two workarounds here that can be used to release or forcefully acquire ownership of a locked cluster.
a. Forcefully acquire cluster ownership.
Warning: This workaround should only be used if the NSX-T manager is not accessible, crashed or is no longer in use and therefore is not able to release cluster ownership.

Note: The following API call should only be used if hosts that are present in the locked cluster are not prepared in any other NSX-T manager in a Multi NSX setup.
  • The NSX manager that wants to acquire the ownership of cluster and it is showing as locked, can use below API to gain ownership of the cluster.
POST https://<NSX-Manager>/api/v1/transport-node-collections?override_nsx_ownership=true

     
b. Clear NSX ownership manually from VC UI:
Warning: This should be done carefully and only if cluster ownership was not removed during TN deletion, and there are no TNs present in the cluster in any NSX manager of Multi NSX setup.

  1. Go to vCenter UI , select the problematic host/cluster - Summary - Custom Attributes.
  2. Select the Custom Attribute starting with "com.vmware.nsx.management.nsxt.****".
  3. Remove the content of the value column present for above custom attribute and click save.
  4. If the value is not removed after the above step, remove the entries from all column's and click on Save.

Note: Don't change any other custom attributes.

Powered by Wolken Software