Import of NSXT Manager as LM fails, when supervisor cluster associated to NSXT manager is disabled or removed

Products

• VMware Cloud Foundation

Issue/Introduction

This issue impacts  enterprise customers when following the below steps:

1. Create a VI domain, deploy Edge Cluster and deploy WCP Supervisor cluster.
2. Now, Disable Supervisor Cluster and supervisor cluster deletion is successful.
3. Deploy NSXT Appliance with Global Manager Role.
4. Add MGMT NSXT as location manager.
5. Add VI NSXT as location manager.
6. After adding NSXT manager as location managers, now try to import entities from VI NSXT manager to global manager. Here import is failing as principal account associated to WCP is not delete. Here is the error message

"Feature PRINCIPAL_IDENTITY-wcp-bab75d6a-7c91-4291-80c6-01ed61308f81 is not supported on Local Manager at site VI."

Symptoms:
Import of NSXT Manager entities fails, as PRINICIPAL ENTITY is not deleted ,when supervisor cluster is disabled or removed.

Environment

VMware Cloud Foundation 4.2
VMware Cloud Foundation 4.3

Resolution

Currently there is no resolution.

Workaround:
To workaround this issue, please follow the below steps:

• Execute below command to get the principle identity ID, which will be used for later steps.

"results" : [ {
"name" : "nsx_policy",
"node_id" : "policy_node",
"is_protected" : true,
"resource_type" : "TokenBasedPrincipalIdentity",
"id" : "8e0a063d-a902-496c-a797-39e3495814dc",
"display_name" : "nsx_policy@policy_node",
"tags" : [ ],
"_create_user" : "nsx_policy",
"_create_time" : 1623718196703,
"_last_modified_user" : "nsx_policy",
"_last_modified_time" : 1623718196703,
"_system_owned" : false,
"_protection" : "REQUIRE_OVERRIDE",
"_revision" : 0
}, {
"name" : "wcp-d76e9193-fedd-4e1b-8211-02b0bd3ca36e",
"node_id" : "wcp-d76e9193-fedd-4e1b-8211-02b0bd3ca36e",
"is_protected" : true,
"resource_type" : "TokenBasedPrincipalIdentity",
"id" : "5134dc22-32e2-451a-9e1b-ec8cf040e376",
"display_name" : "wcp-d76e9193-fedd-4e1b-8211-02b0bd3ca36e",
"description" : "Principal Identity for WCP service",
"tags" : [ ],
"_create_user" : "wcp-d76e9193-fedd-4e1b-8211-02b0bd3ca36e",
"_create_time" : 1623723314100,
"_last_modified_user" : "wcp-d76e9193-fedd-4e1b-8211-02b0bd3ca36e",
"_last_modified_time" : 1623723314100,
"_system_owned" : false,
"_protection" : "REQUIRE_OVERRIDE",
"_revision" : 0
} ]
 

• WCP-scoped PrincipalIdentities are token-based PIs owned by wcpsvc SolutionUser. They are created using JWT token of the wcpsvc solution user, which has the NSX enterprise admin role, and as such can only be deleted by that same user (or possibly another user with same role).
• To delete it manually, follow the below steps:

1. Get the cert & key of the WCP Solution user from VECS:

```
#!/bin/bash
STORE=wcp
ALIAS=wcp
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store ${STORE} --alias ${ALIAS} > certificate.pem
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store ${STORE} --alias ${ALIAS} > private_key.pem

```

2. Get the Holder-of-Key SAML token of the user, using the cert & key retrieved. This can be done by running the following python script in VCSA:

```
import sys
import os
import base64

sys.path.append(os.environ['VMWARE_PYTHON_PATH'])
from pyVim import sso

from cis.cisreglib import LookupServiceClient, VmafdClient

# Get STS URL
ls_url = VmafdClient().get_ls_location()
ls_client = LookupServiceClient(ls_url)

sts_url, sts_cert = ls_client.get_sts_endpoint_data()

sts_auth = sso.SsoAuthenticator(sts_url)
token = sts_auth.get_hok_saml_assertion(
"certificate.pem", "private_key.pem", delegatable=True)

print("\nbase64-encoded: ", base64.b64encode(token.encode()).decode())
 

3. Exchange the HOK SAML token with a JWT token (which is what NSX-T accepts)

dcli +username [email protected] +password 'Admin!23' com vmware vcenter tokenservice tokenexchange exchange --grant-type 'urn:ietf:params:oauth:grant-type:token-exchange' --subject-token-type 'urn:ietf:params:oauth:token-type:saml2' --requested-token-type 'urn:ietf:params:oauth:token-type:id_token' --audience 'vmware-tes:vc:nsxd:nsx' --subject-token ’<base64-encoded SAML>’

4. Use the JWT token in the call to NSX to both GET the list of PIs and DELETE the wcpsvc-scoped PI. Do so by setting the "Authorization: Bearer <jwt token>" header on the NSX API request.

• You can identify the wcpsvc-scoped PI by calling the "/api/v1/trust-management/token-principal-identities" API, and looking for the PI whose description says "Principal Identity for WCP service" as opposed to ones whose description might indicate that they are cluster-scoped PIs (ie: "Principal Identity for WCP cluster....").
• In fact, only proceed to delete the wcpsvc-scoped PI IF AND ONLY IF no other WCP-related PIs are present. Deleting this root wcpsvc-scoped PI when other cluster-scoped PIs may break the system.

curl -kv -XDELETE -H 'Authorization: Bearer <jwt token>' https://<nsx host>/api/v1/trust-management/token-principal-identities/<id of wcpsvc-scoped PI>