Procedure to recover lost minikube data after certificate expiration
Article ID: 398000
Updated On:
Products
Issue/Introduction
Unable to retrieve vcenterprime data we see following error as the minikube data is lost.
kubectl get VCenterPrime -A
No resources found
Environment
TCA 2.3
Cause
Minikube data is lost in following scenario.
1. If the mini kube certificate renewal process was not followed correctly.
2. minikube cert was expired and if TCA-CP is rebooted before renewing the certificate
Resolution
To Recover Minikube Data i.e Management Cluster CRs in TCA 2.3 follow the below steps:
1. SSH to TCA-CP VM and create recovery workspace directory
mkdir -p ~/minikube-data-recovery
2. Recover VCenterPrime Secret
# Set Required Variables
VCENTER_PASSWORD="<vcenterPasswordToBeChanged>"
VCENTER_USERNAME="<vcenterUsernameToBeChanged>"
MGMT_CLUSTER_NAME="<mgmtClusternameToBeChanged>"
cat <<EOF > ~/minikube-data-recovery/vcenterprime-secret.yaml
apiVersion: v1
data:
password: `echo ${VCENTER_USERNAME} |base64 -w 0`
username: `echo ${VCENTER_PASSWORD} |base64 -w 0`
kind: Secret
metadata:
name: vcprime-${MGMT_CLUSTER_NAME}-secret
namespace: tca-system
type: Opaque
EOF
# example Variables:
VCENTER_PASSWORD="password"
VCENTER_USERNAME="[email protected]"
MGMT_CLUSTER_NAME="mc1"
Note: Username and password are intentionally swapped in the secret as per implementation. Do not get confused.
Verify the file ~/minikube-data-recovery/vcenterprime-secret.yaml and kubectl apply it
kubectl apply -f ~/minikube-data-recovery/vcenterprime-secret.yaml --kubeconfig=/home/admin/.kube/config
3. Recover VCenterPrime CR
# Set Required Variables
MGMT_CLUSTER_NAME="<mgmtClusternameToBeChanged>"
VCENTER_HOST="<vcenterFQDNToBeChanged>"
DATACENTER="<dataCenterToBeChanged>"
THUMBPRINT="<thumbprintTobeChanged>"
cat <<EOF > ~/minikube-data-recovery/vcenterprime.yaml
apiVersion: telco.vmware.com/v1alpha1
kind: VCenterPrime
metadata:
name: vcprime-${MGMT_CLUSTER_NAME}
namespace: tca-system
spec:
server:
address: ${VCENTER_HOST}
credentialRef:
kind: Secret
name: vcprime-${MGMT_CLUSTER_NAME}-secret
namespace: tca-system
subConfig:
datacenter: ${DATACENTER}
thumbprint: ${THUMBPRINT}
EOF
# Example Variables:
MGMT_CLUSTER_NAME="mc1"
VCENTER_HOST="vcf-10-158-92-72.lvn.broadcom.net"
DATACENTER="BRCM-TCP"
THUMBPRINT="64:2A:35:41:A5:AE:B5:21:26:45:2F:D9:3F:7B:60:EF:4D:7F:31:35"
Note: you can use the below command to get the sha1 thumbprint of vcenter
openssl s_client -connect ${VCENTER_HOST}:443 </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha1
verify the ~/minikube-data-recovery/vcenterprime-secret.yaml and kubectl apply it
kubectl apply -f ~/minikube-data-recovery/vcenterprime.yaml --kubeconfig=/home/admin/.kube/config
Note: Repeat below steps 4,5,6,7 for all management clusters whose CRs are missing in tca-cp minikube
4. Create Namespace for missing Management Cluster
# Set Variables
MGMT_CLUSTER_NAME="<mgmtClusterNameToBeChanged>"
cat <<EOF > ~/minikube-data-recovery/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
labels:
kubernetes.io/metadata.name: ${MGMT_CLUSTER_NAME}
name: ${MGMT_CLUSTER_NAME}
EOF
# example Variables:
MGMT_CLUSTER_NAME="mc1"
verify ~/minikube-data-recovery/namespace.yaml and apply it
kubectl apply -f ~/minikube-data-recovery/namespace.yaml --kubeconfig=/home/admin/.kube/config
5. Create Management Cluster's kubeconfig secret
MGMT_CLUSTER_NAME="<mgmtClusternameToBeChanged>"
PATH_TO_KUBECONFIG="<pathToKubeconfigToBeChanged>"
cat <<EOF > ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-kubeconfig-secret.yaml
apiVersion: v1
data:
value: `base64 -w 0 ${PATH_TO_KUBECONFIG}`
kind: Secret
metadata:
name: ${MGMT_CLUSTER_NAME}-kubeconfig
namespace: ${MGMT_CLUSTER_NAME}
type: Opaque
EOF
# example Variables:
MGMT_CLUSTER_NAME="mc1"
PATH_TO_KUBECONFIG="/tmp/mc1.kubeconfig"
Note that kubeconfig for mgmt cluster can be copied from /opt/vmware/k8s-bootstrapper/<mgmt_cluster_uuid>/kubeconfig file located on tca-cp. Switch to root for accessing this file and copy to /tmp/ before changing ownership to admin:admin
# Verify the content and apply it
kubectl apply -f ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-kubeconfig-secret.yaml --kubeconfig=/home/admin/.kube/config
6. Create TcaKubernetesCluster CR
# Set Variables
MGMT_CLUSTER_NAME="<mgmtClusternameToBeChanged>"
ENDPOINT_IP="<mgmtclusterendpointIpToBeChanged>"
TBR_BOM="<tbrBomToBeChanged>"
cat <<EOF > ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-tkc.yaml
apiVersion: telco.vmware.com/v1alpha1
kind: TcaKubernetesCluster
metadata:
labels:
telco.vmware.com/cluster-role: management
name: ${MGMT_CLUSTER_NAME}
namespace: ${MGMT_CLUSTER_NAME}
spec:
cloudProviders:
primeRef:
kind: VCenterPrime
name: vprime-${MGMT_CLUSTER_NAME}
namespace: tca-system
clusterNetwork:
pods:
cidrBlocks:
- 100.96.0.0/11
services:
cidrBlocks:
- 100.64.0.0/13
cniType: antrea
controlPlaneEndpoint:
host: ${ENDPOINT_IP}
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
tcaBomReleaseRef:
name: ${TBR_BOM}
EOF
# example Variables:
MGMT_CLUSTER_NAME="mc1"
ENDPOINT_IP="192.168.106.28"
TBR_BOM="tbr-bom-2.3.0-v1.24.10---vmware.1-tkg.2-tca.22936132"
VCENTER_PRIME_NAME="vcprime-mc1"
Note: To get the vcenter prime name you created previously you can use below command
kubectl get vcenterprime -n tca-system --kubeconfig=/home/admin/.kube/config
Note: Please use the below command to find tbr bom version value
kubectl get tbr -n tca-system
Verify the file contents of ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-tkc.yaml and apply it
kubectl apply -f ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-tkc.yaml --kubeconfig=/home/admin/.kube/config
7. Edit Add-Ons from TCA UI And Save
From TCA UI Navigate to the caas cluster → mgmtcluster
Click on Edit Cluster Configuration for the mgmtcluster , do nothing and save
wait for TcaClusterAddon CRs to be recreated in the namespace:
kubectl get TcaClusterAddon -A -w --kubeconfig=/home/admin/.kube/config
Feedback
