TKC cluster creation stuck with ControlPlane node in Provisioned state not able to communicate with TKC API Endpoint

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service

Issue/Introduction

When creating a TKC cluster, for the ControlPlane node to finish to be provisioned, the following connectivity must be enabled:

• From Supervisor ControlPlane nodes to:
  • TKC cluster endpoint.
  • TKC ControlPlane nodes kubernetes endpoints.
• From TKC ControlPlane nodes to:
  • TKC cluster endpoint.
  • TKC ControlPlane nodes kubernetes endpoints.

The purpose of this article is to verify that the above connectivity is enabled and there's nothing blocking it.

Symptoms:

TKC cluster creation gets stuck with ControlPlane nodes in Provisioned state and Worker nodes in Pending state.

ControlPlane node gets an IP address assigned but the cluster creation doesn't progress.

 

Environment

VMware vSphere 7.0 with Tanzu

Cause

The reason why the TKC ControlPlane node got stuck in Provisioned and never progressed to Running state is because the connectivity between TKC ControlPlane node and TKC cluster endpoint was blocked by a firewall.

Resolution

To test the connectivity between the Supervisor nodes, the TKC ControlPlane nodes and the endpoints:

• From Supervisor nodes and from TKC ControlPlane nodes, try to reach the K8s endpoint:
  • curl -k https://<TKC ControlPlane node kubernetes endpoint IP>:6443
• From Supervisor nodes and from TKC ControlPlane nodes, try to reach the TKC endpoint:
  • curl -k https://<TKC cluster endpoint IP>:6443

The endpoints can be obtained as:

• TKC cluster endpoint:
  • kubectl describe tkc -n <namespace> <tkc-name> | grep "API Endpoints" -A2
• TKC ControlPlane node kubernetes endpoint:
  • kubectl describe vm <vm-name> | grep -i "Vm Ip"
  • This is the IP assigned to the TKC ControlPlane VM.

To login to the Supervisor node, refer to the section "How to SSH into Supervisor Control Plane VMs" in the KB Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VMs

To login to TKC ControlPlane node refer to the below document SSH to Tanzu Kubernetes Cluster Nodes as the System User Using a Password

 

The expected output is to get Unauthorized and Forbidden responses for each of the curl commands, meaning the Endpoints are reachable.

E.g.:

 

{

 "kind": "Status",

 "apiVersion": "v1",

 "metadata": {},

 "status": "Failure",

 "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",

 "reason": "Forbidden",

 "details": {},

 "code": 403

}

 

If any of the above curl commands times out, that may mean there's something blocking the connectivity and thus preventing the TKC cluster from being successfully created.

 

Additional Information

Impact/Risks:

Connectivity checks don't have any impacts.