Unable to align or update Tanzu BOM Release (TBR) for Workload Cluster Node Pools after TKG-M Upgrade

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform
• VMware Tanzu Kubernetes Grid

Issue/Introduction

• Workload cluster upgrade from 1.24.x to 1.25.x fails or partially completes.
• The Control Plane successfully transitions to the new version, but Node Pools remain stuck on the legacy TBR (e.g., 1.24.10).
• Attempts to use the "Align TBR" action in the Telco Cloud Automation (TCA) UI fail with validation errors.
• The Upgrade Wizard blocks progression, showing errors such as "unsupported version skew" or refusing the structural payload configuration.
• The parent Tanzu Kubernetes Grid Management (TKG-M) cluster has already been upgraded to a much higher version (e.g., 1.26.8).

Environment

TCA 3.2
TKG 2.5

Cause

This issue is caused by an aggressive lifecycle sequencing mismatch.

TCA Manager implements strict front-end and API-level validation matrices. When a TKG-M management plane is upgraded (e.g., to 1.26.8) while child workload cluster components are still two or more versions behind (e.g., 1.24.x), the active schema definitions lose backward compatibility paths for those specific delta transformations. The UI enforces validation against the current management cluster context, creating a "deadlock" where it cannot reconcile the old worker templates.

Resolution

To resolve this, the infrastructure records must be manually re-aligned within the TCA Manager database to bypass the blocked UI validation mechanism.

Note: Always take a snapshot of the TCA Manager VM and a full database backup before performing manual database modifications.

1. SSH into the TCA Manager appliance as admin and switch to root.
2. Run the following command to create a backup:
   

   kubectl exec -it postgres-0 -n tca-mgr -- pg_dump -U tca_admin tca > /home/admin/tca_db_backup_$(date +%F).sql

3. Access the PostgreSQL prompt:
   

   kubectl exec -it postgres-0 -n tca-mgr -- psql -d tca -U tca_admin

4. Identify the specific row in the K8sClusterDetails table representing the stuck node pool. You can filter by the workload cluster name:
   

   SELECT id, val->>'name' as name, val->'spec'->'tcaBomReleaseRef'->>'name' as tbr FROM "K8sClusterDetails" WHERE val->>'clusterName' = 'YOUR_CLUSTER_NAME';

5. Update the the nodepool TBR BOM in the "K8sClusterDetails" table. (Replace [ID] with the ID found in Step 2 and ensure the tcaBomReleaseRef matches the desired 1.24.10/1.25.x release variables):
   Set the values of TKG_BOM_RELEASE_NAME
   

   UPDATE "K8sClusterDetails" 
   SET val = jsonb_set(
       jsonb_set(
           jsonb_set(val, '{status, phase}', '"Provisioned"', false),
           '{status, needUpgrade}', 'false', false
       ),
       '{spec, tcaBomReleaseRef, name}', '"TKG_BOM_RELEASE_NAME"', false
   ) 
   WHERE id = [ID];

6. Verify that the cluster and node pool status has returned to Active/Provisioned and the TBR versions appear aligned.