Supervisor deactivate request to NSX manager failed in VCF 9.1

Products

• VMware NSX

Issue/Introduction

Supervisor Cluster deactivation in VMware Cloud Foundation (VCF) 9.x fails to complete. The vSphere UI displays the error "Cleanup requests to NSX Manager failed," leaving NSX resources orphaned and the cluster in a "Removing" state.

Symptoms

• vCenter wcpsvc.log contains: NSX cleanup failed: failed to initialize cleanup service com.vmware.vapi.std.errors.unauthorized.
• Manual attempts to delete NSX resources fail with Error code 289 
• You have tried following the resolution steps in a similar KB article: KB 388941, but the manual script execution fails to remove the remaining objects.
• VPC and its associated objects are not deleted as they possess the nsx-operator tag. Consequently, the nsx_policy_cleanup.py script cannot identify or remove them.
• Running GET https://<nsx-manager-ip>/policy/api/v1/orgs/default/projects/default/vpcs/<vpc-id> reveals the following tags:
   

  "tags" : [   { "scope" : "nsx-op/cluster", "tag" : "####" },  { "scope" : "nsx-op/version", "tag" : "1.0.0" },  { "scope" : "nsx-op/namespace", "tag" : "kube-system" },  { "scope" : "nsx/managed-by", "tag" : "nsx-op" }]

Environment

• VMware Cloud Foundation 9.x
• VMware NSX 9.x
• VMware vSphere Kubernetes Service (VKS)

Resolution

If you encounter this issue, please reach out to Broadcom Support.