TCA 2.1- Cluster LCM fails as the Backup & Restore not restoring cluster operator resources present in minikube

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

To provide a workaround steps to avoid backup & restore related CAAS cluster failures like "Error fetching resource Internal Server Error" and "Namespace not found Internal server error" when performing any operation on management cluster & workload clusters.

Symptoms:
When user performs backup and restore from one TCA 2.1 appliance to another, TCA-CP Minikube’s kubeconfig is not updated and Cluster operator resources are not restored which results operations on CaaS Clusters fail.

Environment

VMware Telco Cloud Automation 2.1

Cause

"Error fetching resource Internal Server Error" - Management cluster CRs are not part of the regular backup so when you restore in another machine these CRs will be missing.

"Namespace not found Internal server error" - TCA-CP Minikube's kubeconfig is machine dependant will be different for different machines, kbs kubeconfig has old minikube config reference and when restored it is not updated.

Resolution

VMware is aware of this issue as reported in TCA 2.1, this issue will be address in the next release TCA 2.1.1.

For the existing TCA 2.1 infrastructure, a workaround is available as mentioned below to address the issue.

Workaround:
In order to prevent the above issues following workaround steps has to be applied once per each tca-cp machine before performing any next backup/restore operation in 2.1

These will Update backup process to include management cluster Custom Resource backups.

1. copy the attached backup-and-restore-patch.tar.gz to tca-cp machine

scp backup-and-restore-patch.tar.gz admin@<tca-cp-ip>:~/backup-and-restore-patch.tar.gz

2. extract into /opt/vmware/bin/ folder

ssh admin@<tca-cp-ip> "tar -xvzf ~/backup-and-restore-patch.tar.gz -C /opt/vmware/bin/"

3. make sure permissions and ownerships are preserved

ssh admin@<tca-cp-ip> "cd /opt/vmware/bin/; chown admin:secureall backup.sh restore.sh && chmod 744 backup.sh restore.sh"

After executing the above workaround, backup process includes management cluster CR backups as well.

NOTE: In case if you are restoring to a new tca-cp machine the above workaround has to be applied before performing restore operation.