WCP cluster creation fails with error "Principal 'wcp-cluster-user-xxxx-xxxx' with role '[enterprise_admin]' attempts to delete or modify an object of type nsx$IpAddressPoolIpType it doesn't own."

Products

• VMware NSX

Issue/Introduction

• Customer has Deactivated the Supervisor Cluster
• NSX cleanup failed
• NCP goes in CLBO when supervisor is re-enabled.
• Following is the NCP error message we observe when the WCP cluster creation fails. Verified from NCP pod logs:

kubectl logs –n nsx-system nsx-node-agent-XXXXX –c nsx-node-agent

errorCode="NCP00007"] nsx_ujo.common.utils NSX configuration error: Unexpected error from backend manager (['##.##.##:443', '##.##.##:443', '##.##.##:443', '##.##.##:443']) for PATCH policy/api/v1/infra/ip-pools/domain-c#:######-ip-subnets/#-#-#-#-#: Principal '<wcp-cluster-username_A>' with role '[enterprise_admin]' attempts to delete or modify an object of type nsx$IpAddressPoolIpType it doesn't own. (createUser=wcp-cluster-username_B), allowOverwrite=null); Unexpected error from backend manager (['##.##.##:443', '##.##.##:443', '##.##.##:443', '##.##.##:443']) for PATCH policy/api/v1/infra/ip-pools/domain-c#:######-ippool-#-#-#-#-#-#-#-#/ip-subnets/#-#-#-#-#: Principal 'wcp-cluster-username_A' with role '[enterprise_admin]' attempts to delete or modify an object of type nsx$IpAddressPoolIpType it doesn't own. (createUser=wcp-cluster-username_B, allowOverwrite=null)

Environment

• VMware NSX
• VMware vCenter Server 7.0.x

Cause

If the user reverts the vCenter Server VM snapshot that was taken prior to the supervisor enablement which is not a supported approach for deactivating the supervisor cluster. The result of which was that the IPAM objects created by WCP like IP-pools and IP-Block on the NSX side were not deleted and remained as stale entries in the NSX Database. 

 

Resolution

NOTE: Please make sure there are no subnets under the Pool before attempting the workaround. Can be verified from NSX UI, that the subnet should display as 0 (see below snippet): Networking > IP Address Pools

Steps for workaround:

We need to call the DELETE API for IP-Pool and IP-Block with "X-Allow-Overwrite: true" set in the header. This can be done from Postman or using CURL.

Using Postman tool (Ensure you add header ( 'X-Allow-Overwrite: true') :

DELETE https://<nsx-manager-ip>/policy/api/v1/infra/ip-blocks/<ip-block-id>
DELETE https://<nsx-manager-ip>/policy/api/v1/infra/ip-pools/<ip-pool-id>

OR

using CURL command - 

curl --location --request DELETE 'https://<NSX-MGR-IP>/policy/api/v1/infra/ip-blocks/<IP-block-name>/' \
--header 'X-Allow-Overwrite: true' \
--header 'Content-Type: application/json' \
--data '{}'

curl --location --request DELETE 'https://<NSX-MGR-IP>/policy/api/v1/infra/ip-pools/<IP-pool-name>/' \
--header 'X-Allow-Overwrite: true' \
--header 'Content-Type: application/json' \
--data '{}'

If you believe you have encountered this issue and the noted workaround doesn't resolve the issue, then please open a support case with Broadcom Support, For more information, see Creating and managing Broadcom support cases.