When the client certificate is expired, you will receive the following certificate expiration error when you try to access minikube kubernetes:
kubectl get ns --kubeconfig=/home/admin/.kube/config Unable to connect to the server: x509: certificate has expired or is not yet valid: current time2023-09-14T08:40:21Z is after 2023-03-01T19:17:27Z
Environment
2.x, 2.0, 2.0.1, 2.1.1, 2.1, 2.2, 2.3
Resolution
Issue is resolved in TCA 3.0 as TCA 3.0 does not utilize minikube.
Workaround: Stop the primary TCA services on the TCA-CP (This step only applies to the TCA-CP appliance, skip this step for the TCA Appliance.)
SSH to the TCA CP appliance which contains the expired minikube certificates and switch to root.
Stop TCA web-engine, app-engine and appliance-management services using the following commands:
su -c 'MINIKUBE_HOME=/common/minikube minikube delete' admin
Create a new minikube profile with the following command:
systemctl start minikube
Note: The minikube service can take 2-5 minutes to finish starting up.
Once the minikube service has started successfully check if the cert expired error is gone and check if the AGE of resources listed is new to make sure the minikube has been successfully recreated with the new certificates. Use the following command:
kubectl get ns --kubeconfig=/home/admin/.kube/config
List and note the dependent services of minikube with the following command. Note that these will vary depending on the version and if it is a TCA or a TCA-CP appliance.
systemctl list-dependencies --reverse minikube
Based on the output of the previous command, restart the dependent services in order. As an example, a TCA-CP appliance listed 3 dependent services. The restart commands and order are:
systemctl restart kubecluster-operator.service Note: this will take 2-3 minutes to start systemctl restart hostconfig.service Note: this will take 1-2 minutes to start systemctl restart postgres.service Note: this will take 2-3 minutes to start
Update the /root/.kube/config with the following commands:
Note: the mgmtclusterId is the tkg management cluster id, you can either get this via the CCLI or with the following command: kbsctl show managementclusters
Disable future accidental minikube data deletion during auto certificate renewal with the following command:
sed -i.bak "s/--delete-on-failure=true/--delete-on-failure=false/g" /etc/systemd/minikube-install.sh
Restore minikube data
Restore the minikube data by running the following command:
kubectl apply -f /opt/vmware/k8s-bootstrapper/minikubecr.backup-<YYYYMMDD-HHMMSS> --kubeconfig=/home/admin/.kube/config | tee -a /home/admin/restore_cr-${restore_timestamp}.log
Note: The backup minikube backup file should have been created in the following directory: /opt/vmware/k8s-bootstrapper/restore_timestamp=$(date +"%Y%m%d_%H%M%S")
However, if the output you received when taking the minikube backup differs, please correct the command above with the correct directory path.
Confirm the mgmt cluster namespaces have been successfully restored using the following commands:
kubectl get ns --kubeconfig=/home/admin/.kube/config
kubectl get vcenterprime,tkc,tknp,tka -A --kubeconfig=/home/admin/.kube/config
Start the primary TCA Services
Start the TCA web-engine, app-engine and appliance-management services using the following commands: