Removing a Kubernetes Container Cluster from Cloud Director via the API

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

When trying to delete TKGs (vSphere with Tanzu) clusters that have already been removed from vSphere, from the Kubernetes Containers Plugin in Cloud Director, error messages can be seen in the failed tasks:

• Cluster delete request failed. Please contact your provider if this problem persists. (Error: The request has invalid accept header: Invalid API version requested. Supported API versions are: [38.0.0-alpha, 37.1, 37.0, 36.3, 36.2, 36.1, 36.0, 35.2 [D], 35.0 [D], 34.0 [D], 33.0 [D]] ([D] indicates deprecated versions))
• [ xx ] Cloud Director cannot reach vSphere for Kubernetes, reason message: Could not connect to vSphere for Kubernetes infrastructure. - Could not connect to vSphere for Kubernetes infrastructure. - I/O error on POST request for "https://xxx:443/wcp/login": Connect to xxx:443 [/xxx] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to xxx:443 [/xxx] failed: Connection timed out (Connection timed out) - Connect to xxx:443 [/xxx] failed: Connection timed out (Connection timed out) - Connection timed out (Connection timed out)

Debug Information:

• com.vmware.ssdc.backend.wcp.WcpConnectionPoolException: Cloud Director cannot reach vSphere for Kubernetes, reason message: Could not connect to vSphere for Kubernetes infrastructure.
• at  com.vmware.vcloud.wcp.client.WcpConnectionPool.borrowClient(WcpConnectionPool.java:115)

Cause

This issue occurs if the cluster has been deleted within vSphere itself first.
As Cloud Director did not initiate the operation, it will still contain references to entities that no longer exist.

Resolution

Performing operations on Cloud Director managed objects outside of Cloud Director will generally result data integrity issues between Cloud Director and the other solution.

All actions should be performed via Cloud Director unless explicitly stated by VMware Engineering, Documentation or a Knowledge Base Article.

 

Workaround:

To workaround the issue, you will need to remove the entity definition for the cluster via the Cloud Director Open API.
As the API is based on the Open API, you can leverage the API Explorer to remove the object, or you can use a REST client such as Postman or other.

DELETE 
https://<vcd-fqdn>/cloudapi/1.0.0/entities/{cluster-id}?invokeHooks=false

Note:

• It is very important that we include ?invokeHooks=false to ensure that Cloud Director knows to perform the delete and does not rely on Tanzu to perform an action.
• You can obtain the cluster-id from the Object section of the failed task or via the vApp Details within the Cluster Overview section e.g. urn:vcloud:entity:vmware:capvcdCluster:xxx-xxx-xxx.