TKGm Management Cluster upgrade fails with "failed to get configuration for the with name in-cluster" error

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

Management Cluster upgrade fails with below error:

ipam-in-cluster provider's version is missing in BOM file, so it would not be upgraded
Error: failed to upgrade management cluster providers: failed to apply providers upgrade: failed to get configuration for the  with name in-cluster. Please check the provider name and/or add configuration for new providers using the .clusterctl config file

Cause

The root cause is undetermined.

The issue has been seen after applying KB 374425 - TKGm Management Cluster upgrade fails due to Providers versions mismatch

Resolution

1. From the Management Context:
   # kubectl get provider -A
   
   For example:
   
   $ kubectl get provider -A
NAMESPACE                           NAME                     AGE    TYPE                     PROVIDER      VERSION
caip-in-cluster-system              ipam-in-cluster          3d5h   IPAMProvider             in-cluster    v0.1.0
capi-kubeadm-bootstrap-system       bootstrap-kubeadm        3d5h   BootstrapProvider        kubeadm       v1.4.2
capi-kubeadm-control-plane-system   control-plane-kubeadm    3d5h   ControlPlaneProvider     kubeadm       v1.4.2
capi-system                         cluster-api              3d5h   CoreProvider             cluster-api   v1.4.2
capv-system                         infrastructure-vsphere   3d5h   InfrastructureProvider   vsphere       v1.7.0


2. Take a backup of the "ipam-in-cluster" provider:
   # kubectl get provider -n caip-in-cluster-system ipam-in-cluster -o yaml > ipam-in-cluster.yaml
   
   
3. Manually delete the "ipam-in-cluster" provider:
   # kubectl delete provider -n caip-in-cluster-system ipam-in-cluster


4. Re-trigger the MC upgrade.