Unable to shrink NSX Edges used by WCP in SDDC Manager : Edge cluster <UUID> with selected edge node <edge node uuid> is hosting tier-1 services for kubernetes-workload management.

Products

• VMware NSX

Issue/Introduction

• When trying to remove everything from one cluster to another in the same workload domain. NSX Edge cluster is able to be expanded but cannot shrink.
• Cannot delete edge nodes from old hosts.
• When trying to shrink an NSX Edge cluster that is being used by WCP cluster the following error is presented for both the T1 and any DHCP Servers configured to be used with WCP. 
  
  

  Edge cluster <UUID> with selected edge node <edge node uuid> is hosting tier-1 services for kubernetes-workload management.

  
  
  

   

• Unable to edit the T1 and or DHCP server instance in SDDC Manager. 
  
  

  InvalidRequest .... /edge-nodes/1 can not be deleted as it is being referenced by entity(s):LogicalDhcpServer/.....

  
  
  

  

Environment

NSX

VCF

vSphere with Tanzu

Cause

This is due to WCP/Kubernetes objects being configured to use the selected Edge cluster. WCP configurations are treated as protected objects in NSX and cannot be removed/edited via the UI. This is per design and not a BUG. 

Resolution

The below steps will allow users to attempt to workaround this limitation without performing a "scale-up/scaled-down" of their WCP enviornment. This procedure should be tested in a staging/lab enviornment prior to implementing in a production cluster. 

1. Create a temporary edge-cluster with the same configs as the current active edge nodes. Then gather the below information using API calls with postman:

• Get cluster edge path

  GET https://<nsx-manager-fqdn/api/v1/infra/sites/default/enforcement-points/default/edge-clusters/
  

  • Example:

           {
                "nsx_id": "########-####-####-####-########",
                "inter_site_forwarding_enabled": false,
                "member_node_type": "EDGE_NODE",
                "resource_type": "PolicyEdgeCluster",
                "id": "########-####-####-####-########",
                "display_name": ",Edge-Cluster-Name",
                "tags": [],
                "path": "/infra/sites/default/enforcement-points/default/edge-clusters/########-####-####-####-########",  < -- Copy this path

  • Copy the current edge cluster path (incase a rollback is needed)
• Get Tier 1 ID:
  

  GET https://<nsx-manager-fqdn>/policy/api/v1/infra/tier-1s/

  
  

  • {
        "results": [
            {
                "tier0_path": "/infra/tier-0s/Tier0-name",
                "failover_mode": "NON_PREEMPTIVE",
                "enable_standby_relocation": false,
                ...
                },
                "resource_type": "Tier1",
                "id": "domain-####7:########-####-####-####-####606b817",   <------ Copy this T1 ID
               

• Update Tier-1 to use new "temp" edge cluster path which was collected above using below "patch" command. Set the  "X-Allow-Overwrite=true" header using postman  

  • PATCH:
    
    https://<NSX_FQDN>/policy/api/v1/infra/tier-1s/domain-####7:########-####-####-####-####606b817/locale-services/domain-####7:########-####-####-####-####606b817-0
    
    Body
    {
        "edge_cluster_path": "/infra/sites/default/enforcement-points/default/edge-clusters/########-####-####-####-########"
    }

• At this point, attempt to shrink the edge cluster once again. Keep in mind there may be other configurations attached to this edge cluster. If there are DHCP servers in-use and pointing to the old edge cluster, shrinking the edge cluster will not be possible and an error will be displayed as shown in the "introduction" section of this article. 
• Next steps will be to remove/delete the DHCP server setup (THIS IS A POTENTIALLY DISRUPTIVE OPERATION). All pods set to use the DHCP servers will lose IP config. Again these steps should be tested to determine how it will affect workloads present in the enviornment. 
• Once the DHCP servers are removed, re-attempt to shrink the edge cluster in SDDC manager.