Recover lost minikube data after certificate expiration

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

•  Unable to retrieve vcenterprime data we see following error as the minikube data is lost

kubectl get VCenterPrime -A

No resources found 

Environment

TCA: 2.2, 2.3

TCP: 2.7, 3.0

Cause

Minikube data is lost in following scenarios

• The mini kube certificate renewal process was not followed correctly.
• The minikube cert was expired and if TCA-CP is rebooted before renewing the certificate.

Resolution

Recover the minikube data i.e management cluster custom resources

1. SSH to TCA-CP VM  and create recovery workspace directory
   

   mkdir -p ~/minikube-data-recovery

2. Recover VCenterPrime Secret
   

   # Set Required Variables
VCENTER_PASSWORD="<vcenterPasswordToBeChanged>"
VCENTER_USERNAME="<vcenterUsernameToBeChanged>"
MGMT_CLUSTER_NAME="<mgmtClusternameToBeChanged>"

   cat <<EOF > ~/minikube-data-recovery/vcenterprime-secret.yaml
   apiVersion: v1
data:
  password: `echo ${VCENTER_USERNAME} |base64 -w 0`
  username: `echo ${VCENTER_PASSWORD} |base64 -w 0`
kind: Secret
metadata:
  name: vcprime-${MGMT_CLUSTER_NAME}-secret
  namespace: tca-system
type: Opaque
EOF

   • Note: Username and password are intentionally swapped in the secret as per implementation. Do not get confused.

3. Verify the file ~/minikube-data-recovery/vcenterprime-secret.yaml and kubectl apply it
   kubectl apply -f ~/minikube-data-recovery/vcenterprime-secret.yaml  --kubeconfig=/home/admin/.kube/config
   
   
4. Recover VCenterPrime CR
   

   # Set Required Variables
MGMT_CLUSTER_NAME="<mgmtClusternameToBeChanged>"
VCENTER_HOST="<vcenterFQDNToBeChanged>"
DATACENTER="<dataCenterToBeChanged>"
THUMBPRINT="<thumbprintTobeChanged>"

   cat <<EOF > ~/minikube-data-recovery/vcenterprime.yaml
apiVersion: telco.vmware.com/v1alpha1
kind: VCenterPrime
metadata:
  name: vcprime-${MGMT_CLUSTER_NAME}
  namespace: tca-system
spec:
  server:
    address: ${VCENTER_HOST}
    credentialRef:
      kind: Secret
      name: vcprime-${MGMT_CLUSTER_NAME}-secret
      namespace: tca-system
  subConfig:
    datacenter: ${DATACENTER}
    thumbprint: ${THUMBPRINT}
EOF

   • Note: you can use the below command to get the sha1 thumbprint of vcenter
     openssl s_client -connect ${VCENTER_HOST}:443 </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha1

5. Verify the ~/minikube-data-recovery/vcenterprime-secret.yaml and kubectl apply it
   kubectl apply -f ~/minikube-data-recovery/vcenterprime.yaml --kubeconfig=/home/admin/.kube/config

   • Note: Repeat below steps 4,5,6,7 for all management clusters whose CRs are missing in tca-cp minikube

6. Create Namespace for missing Management Cluster
   
   # Set Variables
MGMT_CLUSTER_NAME="<mgmtClusterNameToBeChanged>"

   cat <<EOF > ~/minikube-data-recovery/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: ${MGMT_CLUSTER_NAME}
  name: ${MGMT_CLUSTER_NAME}
   EOF

7. verify ~/minikube-data-recovery/namespace.yaml and apply it
   kubectl apply -f ~/minikube-data-recovery/namespace.yaml --kubeconfig=/home/admin/.kube/config

8. Create Management Cluster's kubeconfig secret 

   MGMT_CLUSTER_NAME="<mgmtClusternameToBeChanged>"
PATH_TO_KUBECONFIG="<pathToKubeconfigToBeChanged>"

   cat <<EOF > ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-kubeconfig-secret.yaml
apiVersion: v1
data:
  value: `base64 -w 0 ${PATH_TO_KUBECONFIG}`
kind: Secret
metadata:
  name: ${MGMT_CLUSTER_NAME}-kubeconfig
  namespace: ${MGMT_CLUSTER_NAME}
type: Opaque
EOF

   • Note: The kubeconfig for mgmt cluster can be copied from /opt/vmware/k8s-bootstrapper/<mgmt_cluster_uuid>/kubeconfig file located on tca-cp.

9. Switch to root for accessing this file and copy to /tmp/ before changing ownership to admin:admin

10. Verify the content and apply it
    kubectl apply -f ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-kubeconfig-secret.yaml --kubeconfig=/home/admin/.kube/config

11. Create TcaKubernetesCluster CR
    
    # Set Variables
MGMT_CLUSTER_NAME="<mgmtClusternameToBeChanged>"
ENDPOINT_IP="<mgmtclusterendpointIpToBeChanged>"
TBR_BOM="<tbrBomToBeChanged>"

    cat <<EOF > ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-tkc.yaml
apiVersion: telco.vmware.com/v1alpha1
kind: TcaKubernetesCluster
metadata:
  labels:
    telco.vmware.com/cluster-role: management
  name: ${MGMT_CLUSTER_NAME}
  namespace: ${MGMT_CLUSTER_NAME}
spec:
  cloudProviders:
    primeRef:
      kind: VCenterPrime
      name: vprime-${MGMT_CLUSTER_NAME}
      namespace: tca-system
  clusterNetwork:
    pods:
      cidrBlocks:
      - 100.96.0.0/11
    services:
      cidrBlocks:
      - 100.64.0.0/13
  cniType: antrea
  controlPlaneEndpoint:
    host: ${ENDPOINT_IP}
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  tcaBomReleaseRef:
    name: ${TBR_BOM}
EOF

    • Note: To get the vcenter prime name you created previously you can use below command 

      kubectl get vcenterprime -n tca-system --kubeconfig=/home/admin/.kube/config

    • Note: Please use the below command to find tbr bom version value

      kubectl get tbr -n tca-system

12. Verify the file contents of ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-tkc.yaml and apply it 

    kubectl apply -f ~/minikube-data-recovery/${MGMT_CLUSTER_NAME}-tkc.yaml --kubeconfig=/home/admin/.kube/config

13. Edit Add-Ons from TCA UI And Save 

    1. From TCA UI Navigate to the caas cluster → mgmtcluster

    2. Click on Edit Cluster Configuration for the mgmtcluster , do nothing and save 

14. Wait for TcaClusterAddon CRs to be recreated in the namespace:

    kubectl get TcaClusterAddon -A -w --kubeconfig=/home/admin/.kube/config

Additional Information

Manually Creating a TcaNodePool (tknp)