How to renew Minikube certificates

Products

• VMware Telco Cloud Automation

Issue/Introduction

Symptoms:

When the client certificate is expired, you will receive the following certificate expiration error when you try to access minikube kubernetes:

kubectl get ns --kubeconfig=/home/admin/.kube/config
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-09-14T08:40:21Z is after 2023-03-01T19:17:27Z

Environment

2.x, 2.0, 2.0.1, 2.1.1, 2.1, 2.2, 2.3

Resolution

Issue is resolved in TCA 3.0 as TCA 3.0 does not utilize minikube.

Workaround:
Stop the primary TCA services on the TCA-CP (This step only applies to the TCA-CP appliance, skip this step for the TCA Appliance.)

1. SSH to the TCA CP appliance which contains the expired minikube certificates and switch to root.
2. Stop TCA web-engine, app-engine and appliance-management services using the following commands:

systemctl stop appliance-management
systemctl stop web-engine
systemctl stop app-engine


Backup existing minikube data on the TCA-CP (This step only applies to the TCA-CP appliance, skip this step for the TCA appliance.)

1. SSH to the TCA CP appliance where minikube certificates are expired and switch to root.
2. Download and copy the minikube-data-backup.sh script into the /home/admin/ directory.
3. Execute the script to backup the existing minikube data (i.e Custom Resources)
   using the following commands:

         chmod +x /home/admin/minikube-data-backup.sh
    bash /home/admin/minikube-data-backup.sh

Note: Take note of the location for the backup files from the output to restore the minikube data after the certificates have been renewed.


Renew minikube expired certificates (This step applies to both the TCA and TCA-CP appliances.)

1. Stop the minikube service using the following command:

systemctl stop minikube

2. Take a backup of the minikube configuration folder using the following command:

cp -rp /common/minikube/.minikube /common/minikube/minikube-conf-backup-`date +"%Y%m%d_%H%M%S"`

3. Delete the existing profile with the admin user

su -c 'MINIKUBE_HOME=/common/minikube minikube delete' admin

4. Create a new minikube profile with the following command:

systemctl start minikube

Note: The minikube service can take 2-5 minutes to finish starting up.

5. Once the minikube service has started successfully check if the cert expired error is gone and check if the AGE of resources listed is new to make sure the minikube has been successfully recreated with the new certificates.  Use the following command:

kubectl get ns --kubeconfig=/home/admin/.kube/config

6. List and note the dependent services of minikube with the following command.  Note that these will vary depending on the version and if it is a TCA or a TCA-CP appliance. 

systemctl list-dependencies --reverse minikube

7. Based on the output of the previous command, restart the dependent services in order.  As an example, a TCA-CP appliance listed 3 dependent services.  The restart commands and order are:  

systemctl restart kubecluster-operator.service
Note: this will take 2-3 minutes to start
systemctl restart hostconfig.service
Note: this will take 1-2 minutes to start
systemctl restart postgres.service
Note: this will take 2-3 minutes to start

8. Update the /root/.kube/config with the following commands:

cp /root/.kube/config /root/.kube/config.bak
KUBECONFIG=/home/admin/.kube/config:/root/.kube/config.bak
kubectl config view --flatten > /root/.kube/config

9. Restore the missing management cluster contexts with the following commands: 
   • root_kubeconfig="/root/.kube/config.bak-`date +"%Y%m%d_%H%M%S"`"
   • cp /root/.kube/config ${root_kubeconfig}
   • mgmtcluster_kubeconfig="/opt/vmware/k8s-bootstrapper/<mgmtclusterId>/kubeconfig"KUBECONFIG=${mgmtcluster_kubeconfig}:${root_kubeconfig} kubectl config view --flatten > /root/.kube/config
     • Note: the mgmtclusterId is the tkg management cluster id, you can either get this via the CCLI or with the following command:  kbsctl show managementclusters
10. Disable future accidental minikube data deletion during auto certificate renewal with the following command:

sed -i.bak "s/--delete-on-failure=true/--delete-on-failure=false/g" /etc/systemd/minikube-install.sh

Restore minikube data

1. Restore the minikube data by running the following command:

kubectl apply -f /opt/vmware/k8s-bootstrapper/minikubecr.backup-<YYYYMMDD-HHMMSS> --kubeconfig=/home/admin/.kube/config | tee -a /home/admin/restore_cr-${restore_timestamp}.log

Note: The backup minikube backup file should have been created in the following directory:
/opt/vmware/k8s-bootstrapper/restore_timestamp=$(date +"%Y%m%d_%H%M%S")

However, if the output you received when taking the minikube backup differs, please correct the command above with the correct directory path.

2. Confirm the mgmt cluster namespaces have been successfully restored using the following commands:

• kubectl get ns --kubeconfig=/home/admin/.kube/config
• kubectl get vcenterprime,tkc,tknp,tka -A  --kubeconfig=/home/admin/.kube/config

Start the primary TCA Services

1. Start the TCA web-engine, app-engine and appliance-management services using the following commands:

systemctl start appliance-management
systemctl start web-engine
systemctl start app-engine

Additional Information

The minikube certificates are valid for 1 year in TCA 2.1.x and for 3 years in TCA 2.3.x.