Management Cluster upgrade fails with error "only support to upgrade the management cluster to []" in TCA

Products

• VMware Telco Cloud Automation

Issue/Introduction

When attempting to upgrade a VMware Telco Cloud Automation (TCA) management cluster, the operation fails. The task log shows an HTTP 400 Bad Request response with the error message:

{"code":"bad_request","message":"only support to upgrade the management cluster to []"}.

Analysis of the kbs-tkg2xx pod logs (k8s bootstrapper service) reveals a corresponding error:

[Err-common]: not find default tanzu kubernetes release file in ~/.config/tanzu/tkg/bom , which immediately precedes the HTTP 400 bad_request log entry.

Environment

3.x

Cause

The root cause of this failure is the absence of the Tanzu Kubernetes release BOM files in the /root/.config/tanzu/tkg/ directory within the kbs-tkg2xx pod. The k8s-bootstrapper-service requires these files to determine the valid upgrade paths. When these files are missing, the service cannot find any supported versions, resulting in the empty list [] reported in the error message.

Resolution

To resolve this issue, you must restore the missing TKG configuration and BOM files from an archive located inside the kbs-tkg220 pod.

1. Open an SSH session to the TCA-CP appliance that manages the cluster.
2. Find the full name of the kbs-tkg220 pod and exec into it:

kubectl exec -it $(kubectl get po -n tca-cp-cn | grep kbs-tkg220 | awk '{print $1}') -n tca-cp-cn -- /bin/bash

    3. You can verify the issue by listing the contents of the directory, which will likely be empty:

ls /root/.config/tanzu/tkg

    4. From the pod's shell (e.g., in the / directory), run the following command to re-extract the configuration files from the local archive:

tar xvfz /tkgcfg.tar.gz -C /root/.config/tanzu

   5. After the files are restored, retry the management cluster upgrade from the TCA UI.

Workaround (Optional): If upgrade requests are stuck, you can optionally restart the tca-app pods on the TCA Manager and TCA-CP appliances after performing the steps above:

kubectl rollout restart deployment -n tca-cp-cn tca-app