TCA Add-on Reconciliation Failure

Products

VMware Telco Cloud Automation

Issue/Introduction

In a VMware Telco Cloud Automation (TCA) environment, users may encounter issues where certain add-ons fail to reconcile.

Running the command and describe failed add-on :

kubectl get tka -A
kubectl describe tka -n namespace addon_name

Describing add-on we can see this message printed :

PreCheck failed" TcaClusterAddon="wk1-cluster/harbor-tca-addon" TcaKubernetesCluster.telco.vmware.com "whereabouts" not found
TcaKubeControlPlane is not ready, requeuing this reconcile

PKGI or APPS packages may fail to reconcile due to an API request failure, leading to deployment and upgrade issues. When running the command:

kubectl get pkgi -A
kubectl describe pkgi pkgi_package -n namespace

Example of the failure :

tca-system   tca-kubecluster-operator   tca-kubecluster-operator.telco.vmware.com   2.3.0-21465265   Reconcile failed: the server is currently unable to handle the request (get pack...)

This prevents the successful provisioning and operation of required add-ons within the cluster.

Environment

2.x , 3.x

Cause

This issue is typically caused by a failure in the reconciliation process due to:

TCA Components Not Responding Properly: The kapp-controller or tca-kubecluster-operator pod may be in an unhealthy state.
API Server Limitations: The system may be temporarily unable to process requests due to excessive load or API resource unavailability.
Corrupted or Stuck Add-on Deployments: Add-ons may be stuck in a pending or failed state, preventing further updates.

Resolution

To resolve this issue, restart the necessary TCA components to force reconciliation and restore add-on functionality:

kubectl get pkgi -A
kubectl get apps -A

Identify the add-ons that are failing reconciliation.

Restart the kapp-controller and tca-kubecluster-operator Pods:

kubectl delete pod -n tca-system -l app=kapp-controller
kubectl delete pod -n tca-system -l app=tca-kubecluster-operator

This will terminate and restart the affected pods.

Verify if Reconciliation is Successful:

kubectl get pkgi -A
kubectl get apps -A

Check the Status of the Failing addon:

Get the add-on list from the all clusters :

kubectl get tka -A

Describe add-on

kubectl describe tka addon -n namespace

Restarting the kapp-controller and tca-kubecluster-operator pods typically resolves reconciliation failures in TCA. If the issue persists, consider checking for connectivity issues, resource limitations, or additional service failures within the cluster.