Cluster rehome task fails or gets stuck in pending state
search cancel

Cluster rehome task fails or gets stuck in pending state

book

Article ID: 441337

calendar_today

Updated On:

Products

VMware Telco Cloud Automation VMware Telco Cloud Platform

Issue/Introduction

  • Rehome operation fails, does not complete, or hangs indefinitely.
  • Attempting to move a workload cluster from a source management cluster to a destination management cluster (e.g., during an upgrade procedure) fails

Environment

TCA: 3.2

TCP: 5.0

Cause

The destination management cluster may not be fully healthy or the underlying infrastructure may contain some errors that the destination management cluster cannot fully accept the ownership of the workload cluster.

Resolution

  • Search for the workload cluster custom resources on the source management cluster:

    kubectl get tkc,tkcp,tknp,tka -n <WORKLOAD_CLUSTER_NAME>

    • If the custom resources are still present on the source management cluster,  the rehome did not migrate them to the target. Apply KB Workload cluster rehoming operation stuck due to failed destination management cluster to revert the failed re-home operation and point workload cluster to source management cluster. . 
    • If the custom resources are no longer present on the source management cluster, the rehome job completed successfully but other issues on the target management cluster are prohibiting the task form completing.

      1. On the target management cluster, check the status of the workload cluster custom resources:

        kubectl get tkc,tkcp,tknp,tka -n <WORKLOAD_CLUSTER_NAME>

      2. Identify any custom resources that are not in a Provisioned status and inspect for errors:

        kubectl get <KIND> <NAME> -n <WORKLOAD_CLUSTER_NAME> -o yaml

      • Resolve any errors that were identified within the respective yaml file and confirm the cluster is in a Provisioned state. This will allow the rehome task to complete.

      • If the errors are unclear or cannot be resolved directly, collect support bundles from both the source and target management clusters and Contact Broadcom support.

Additional Information

Collecting diagnostic information for Telco Cloud Platform cases