A guest cluster's KubeAdmControlPlane (KCP) got deleted with no backup

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

In vSphere Kubernetes Service, a guest cluster has it's KubeAdmControlPlane (KCP) deleted, there is no KubeAdmControlPlane (KCP) backup.

This delete was an unsupported action carried out from a Supervisor Control Plane node.

The KubeadmControlPlane (KCP) is a custom resource that is provided by the Kubernetes Cluster API, and that allows to manage a set of Machines hosting control plane nodes created with kubeadm.

If KCP is deleted this can have a catastrophic effect on the guest cluster it manages.

Environment

vSphere Kubernetes Service (VKS) 8.0 +

Guest cluster 1.30.1

Cause

The KCP will not be recreated. Deletion of KubeAdmControlPlane (KCP) is not a supported action and must not be done.

Deletion of the KCP is not a recommended action and doing so will effectively destroy the guest cluster and leave it unusable.

PLEASE NOTE: When on the Supervisor Control Plane VM you have permissions to permanently damage the cluster. If VMware Support finds evidence of a customer making changes to the Supervisor Cluster from the Supervisor Control Plane VM, they may mark your cluster as unsupported and require you redeploy the entire vSphere Kubernetes Service solution. Only use this session to test networks, look at logs, and run kubectl logs/get/describe commands. Do not deploy, delete, or edit anything from this session without the express permission of a KB or VMware by Broadcom Technical Support.

 

Resolution

If a backup of the KCP exists;

 - Restore the KCP backup
 - After restore using Velero the guest cluster should become usable.

Link to Velero Backup / Restore guides are found here.

If a backup does not exist:

 - Delete the affected guest cluster.
 - Create a new KubeAdmControlPlane (KCP) and guest cluster.
 - Move workload to the new cluster

Additional Information

Never delete KCP of a guest cluster.
Deleting the KubeAdmControlPlane (KCP) of a guest cluster will effectively make a guest cluster unusable.