Cluster re-home fails with error [CAS111505] failed to get object graph after TKG cluster upgrade

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

When attempting to move a workload cluster from a source management cluster to a target cluster after an upgrade (e.g., from v1.24 to v1.27.15), the re-home process fails with the following

Error: [CAS111505] failed to get object graph: failed to check virtual node: Missing Machine [Namespace]/[Machine_Name] in source management cluster which listed in ownerReferences of object VSphereMachine [VSphereMachine_Name]

Environment

TCA: 3.x, TCP: 5.x

Cause

The failure is caused by stale Machine and VSphereVM objects remaining in the management cluster. The VSphereMachine object incorrectly references a Machine object that was deleted during the upgrade or previous cluster operations. This reference mismatch prevents the object graph from being validated during the re-home process.

Resolution

To resolve this issue, manually clean up the leftover references in the management cluster:

1. Identify Stale Objects: Connect to the management cluster and check for orphaned resources associated with the stuck workload cluster:

   kubectl get ma,vspheremachine,vspherevms -A | grep <workload-cluster-name>

2. Remove Finalizers (if necessary): If the objects are stuck in a deleting state, remove their finalizers to allow garbage collection:

    

   kubectl patch vspheremachine <name> -n <namespace> --type merge -p '{"metadata":{"finalizers":[]}}'kubectl patch vspherevm <name> -n <namespace> --type merge -p '{"metadata":{"finalizers":[]}}'kubectl patch machine <name> -n <namespace> --type merge -p '{"metadata":{"finalizers":[]}}'

3. Delete Orphaned Objects: Manually delete the identified stale objects:

    

   kubectl delete vspheremachine <name> -n <namespace>kubectl delete vspherevm <name> -n <namespace>kubectl delete machine <name> -n <namespace>

4. Verify Cleanup: Ensure no stale references remain:

    

   kubectl get ma,vspheremachine,vspherevms -A | grep <workload-cluster-name>

5. Retry Re-home if it doesn't auto-reconcile: Restart the re-home operation from the Telco Cloud Automation (TCA) UI or Workflow Hub.

Additional Information

Deleted k8s node is not recreated by Machine Health Check (MHC) in TKGm