TMC Cluster upgrade fails after vSphere upgrade

Products

• Tanzu Mission Control

Issue/Introduction

Symptoms:

Upgrading TKC cluster on TMC may get stuck with the following error:

API Error: Failed to upgrade cluster: (target=mc:XX, intentId=XX): Operation cannot be fulfilled on tanzukubernetesclusters.run.tanzu.vmware.com "XX": the object has been modified; please apply your changes to the latest version and try again (failed precondition)"

 

Cause

This API error means the data in TMC database and management cluster are not consistent.

Resolution

 

 

 

Workaround:
You may inspect the logs for the following pods in the TMC namespace

kubectl -n vmware-system-tmc logs -l "app=agent-updater" > agent-updater.log
kubectl -n vmware-system-tmc describe pod -l "app=agent-updater" > agent-updater-describe.log
kubectl -n vmware-system-tmc logs -l "app=sync-agent" > sync-agent.log
kubectl -n vmware-system-tmc describe pod -l "app=sync-agent" > sync-agent-describe.log

If those pods are not able to sync-up with the TMC database, you can recreate them

kubectl -n vmware-system-tmc delete pod -l "app=agent-updater"
kubectl -n vmware-system-tmc delete pod -l "app=sync-agent"

Additional Information

Impact/Risks:

Upgrading the k8s clusters from TMC UI cannot be triggered.