Error: java.lang.ClassCastException during TKG Management Cluster addon upgrade in VMware Telco Cloud Automation 3.4

Products

• VMware Telco Cloud Automation
• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• When upgrading a Tanzu Kubernetes Grid Management (TKG-M) cluster from version 1.28.x to 1.29.x in VMware Telco Cloud Automation (TCA) 3.4, the addon upgrade phase fails.
• The management cluster becomes unmanageable, and options in the TCA UI appear greyed out.

Environment

• VMware Telco Cloud Automation (TCA) 3.4.0.1
• Tanzu Kubernetes Grid Management (TKG-M) 2.5.4

Cause

• Addon upgrade phase fails with error: INTERNAL_SERVER_ERROR Failed to update addon for cluster. Error: class java.lang.String cannot be cast to class java.util.Map.
• Management cluster status in TCA UI is suspended or options are greyed out.
• Nodes show version 1.29.15, but LCM operations are blocked.
• The kubeconfig-secret in the tca-cp-cn namespace on the TCA Control Plane becomes empty (contains "" instead of a base64 string). 

Resolution

To resolve the issue in existing 3.4 environments:

1. Log in to the TCA-CP VM via SSH as admin and switch to root: sudo su -


2. Generate a Base64 encoded string of the valid local kubeconfig: cat /etc/kubernetes/admin.conf | base64 -w0
   
   
3. Edit the kubeconfig-secret in the tca-cp-cn namespace: 
   

   kubectl edit secret kubeconfig-secret -n tca-cp-cn

4. Replace the empty kubeconfig: value under the data: section with the string copied in step 2.
   
   
5. Save and exit (:wq).
   
   
6. Restart the application services: 
   

   1.  TCA-CP
   # root user:
   kubectl rollout restart deployment -n  tca-cp-cn tca-app
   kubectl rollout restart deployment -n  tca-cp-cn tca-api
   
   
   2. TCA-M
   # root user:
   kubectl rollout restart deployment -n tca-mgr tca-app
   kubectl rollout restart deployment -n tca-mgr tca-api
   
   3. check status
   
   kubectl get pods -n  tca-cp-cn  | grep tca-app
   kubectl get pods -n  tca-cp-cn  | grep tca-api
   
   kubectl get pods -n  tca-mgr  | grep tca-app
   kubectl get pods -n  tca-mgr  | grep tca-api

7. Manually set the cluster status to ACTIVE in the TCA-M Database if it remains suspended: 
   

   To connect TCA-M DB:
   kubectl exec -it postgres-0 -n tca-mgr -- psql -d tca -U tca_admin -h localhost
   
   List the management cluster:
   select id, val->>'rowType', val->'metadata'->>'name' from "K8sClusterDetails";
    
   Run command to make sure the right management cluster is returned: 
   select * from  "K8sClusterDetails" where id=xxx;
   
   Update the status of the management cluster:
   update "K8sClusterDetails" set val = jsonb_set(val, '{status}', '"ACTIVE"') where id=xxx;