TCA 2.1.0 - CapbkBootstrapTokenTtl in kbs.conf

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

How to modifiy CapbkBootstrapTokenTtl in kbs.conf for cluster control plane node cannot join the first control plane node.

Symptoms:

• When a user creates a cluster with multiple replicas for the control plane node, the first control plane node is created successfully. However the remaining control plane fails to join the first control plane and cluster creation times out.
• On the second control plane node the /var/log/cloud-init-output.log, file contains errors similar to:

"error execution phase control-plane-join/mark-control-plane: couldn't validate the identity of the API Server: could not find a JWS signature in the cluster-info ConfigMap for token ID"
 

Environment

VMware Telco Cloud Automation 2.0.1
VMware Telco Cloud Automation 2.0

Cause

This can occur when the second node takes more then 15 minutes to join as internally there is a ttl token which times out.

Resolution

VMware is aware of this issue as reported in TCA 2.1, current workaround is available as mentioned below to address the issue.

TKG 1.7 release will have a permanent fix ensuring TTL gets extended automatically until a node successfully joins.

Workaround:

Based the log message's timestamp, user can see how long the kubeadm join takes, then user need to perform the following:

1. ssh into tca-bootstrapper vm in cloud native if the cluster timeout is creating TCA cluster 

or  

ssh into tca-cp if cluster timeout is during CAAS workflow

2. change /opt/vmware/k8s-bootstrapper/kbs.conf to add

[AdvancedConf]
CapbkBootstrapTokenTtl = xx

xx is an interger number larger than 15, the unit is in minute

3. Restart bootstrapper service using the following command:
 

systemctl restart bootstrapperd

NOTE: Post making the above changes ensure that you recreate the Control Plane Management Cluster.

Additionally in case if we need to Modify the existing Management Cluster without recreation follow the steps as mentioned below:

Step 1: Validate the configuration: 

 

kubectl get pods capi-kubeadm-bootstrap-controller-manager-XXXXXXXXXX -n capi-kubeadm-bootstrap-system -o yaml 

spec:
  containers:
  - args:
    - --leader-elect
    - --metrics-bind-addr=localhost:8080
    - --feature-gates=MachinePool=false
    - --bootstrap-token-ttl=90m

Step 2: Modify the existing Management Cluster (Live/Runtime) by updating the "bootstrap-token-ttl" using the following command: 
 

kubectl edit deployment capi-kubeadm-bootstrap-controller-manager -n capi-kubeadm-bootstrap-system