Control plane node stuck in provisioned status

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

Control plane node stuck in "provisioned" status as it cannot join the ETCD cluster as a member

Environment

TCA: 3.2, TCP: 5.0

Cause

The new node is blocked from joining etcd because an older node in the cluster has an expired certificate, breaking peer-to-peer communication. You can verify by connecting to the node directly as capv from TCA-CP (root) and checking the cloud-init-output.log

While connected to the VIP, identify the node the old control plane node that is not a ETCD follower using the below commands

sudo -i

alias etcdctl='/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/*/fs/usr/local/bin/etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key --cacert /etc/kubernetes/pki/etcd/ca.crt'

etcdctl -w table member list

Resolution

Once the node is identified delete the machine, vspheremachines and vspherevm object from the management cluster

1. SSH to the management cluster

2. Delete the old control plane node that is not a ETCD member by deleting the aforementioned object

3. Delete the control plane node that is in provisioned state

This will start a new control plane node deployment process and resolve the issue

Additional Information

How to Defragment a TKGm etcd Cluster 
Manually Renew Cluster Certificates