Workload cluster upgrade stuck without any progress

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Workload cluster upgrade does not proceed further in the TCA UI. 
• The workload cluster moving status (under Infrastructure > CaaS Infrastructure > Cluster Instances, checking the Configuration and Control Plane tab ) shows as not ready in the UI.
• When we check the workload cluster moving status using the API explorer it shows as complete. 

1. 1. Log in to the VMware Telco Cloud Automation Manager web console.
   2. Click on the User Profile icon (usually your username or a person icon) in the top-right corner.
   3. Select API Explorer from the dropdown menu.
   4. In the API Explorer, you will see different API definitions. For cluster-specific operations, look for the CaaS (Container-as-a-Service) or Tanzu Kubernetes Grid API segments.

 

Environment

3.x

Cause

There is discrepancy for the management cluster name between the metadata in the database and in the TCA UI.

Resolution

Note : Please make sure you take a snapshot of the TCA Manager Virtual machine and also a back up of the TCA database  or the TCA application back up from the UI as outlined below before making any changes.

TCA Database Backup using CLI

• 1. SSH into the TCA Manager as admin and switch to root.

  2. Execute backup file of the tca database:
     kubectl exec -it postgres-0 -n tca-mgr -- pg_dump -U tca_admin tca > /home/admin/tca_db_backup_$(date +%F).sql

  3. Verify the backup file: Check that the file was created and has a size greater than 0:
     ls -lh /home/admin/tca_db_backup_*.sql

TCA Application back up from the UI

See Backing Up VMware Telco Cloud Automation Control Plane

Update the correct management cluster name in the TCA database  

1. SSH into the TCA Manager as admin and run the below command to access the TCA database
   
   kubectl exec -it postgres-0 -n tca-mgr -- psql -d tca -U tca_admin
   
   
2. Check the current management cluster name in the metadata associated with the workload cluster tables.
   
   SELECT val->'metadata'->>'name' as cluster_name, val->'status'->'currentMgmtClusterName' as current_mgmt FROM public."K8sClusterDetails" WHERE val->>'rowType'='cluster' AND val->'metadata'->>'name'='workload cluster name';
   
   
3. Verify if the management cluster name in the TCA UI and in the metadata are same. If not, update the correct name using the below query.
   
   UPDATE public."K8sClusterDetails" SET val = jsonb_set(val, '{status, currentMgmtClusterName}', '"<correct management cluster name>"') where val->>'rowType'='cluster' and val->'metadata'->>'name'='<workload cluster name>';