Management Cluster Upgrade Failed To Find Core Provider
search cancel

Management Cluster Upgrade Failed To Find Core Provider

book

Article ID: 317449

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

Symptoms:

  • While attempting to upgrade a Tanzu Kubernetes Grid (TKG) management cluster version from 1.3.1 to 1.4.0 users see similar to the following in the tanzu CLI output after the first upgrade attempt: 
    • updating additional components: 'addons-management/tanzu-addons-manager' ...
updating additional components: 'tkr/tkr-controller' ...
updating additional components: 'addons-management/core-package-repo' ...
Error: failed to deploy additional components after kubernetes upgrade: error while upgrading additional component 'addons-management/standard-package-repo': kubectl apply failed, output: secret/mgmt-cluster-standard-package-repository-crs configured
Error from server (InternalError): error when creating "/tmp/kubeapply-211654953": Internal error occurred: failed calling webhook "default.clusterresourceset.addons.cluster.x-k8s.io": Post "https://capi-webhook-service.capi-webhook-system.svc:443/mutate-addons-cluster-x-k8s-io-v1alpha3-clusterresourceset?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
: exit status 1
  • The following messages are seen in the cert-manager-webhook pod logs: 
    • W1001 12:15:35.653078       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1001 12:15:35.653294       1 webhook.go:57] cert-manager/webhook "msg"="using dynamic certificate generating using CA stored in Secret resource"  "secret_name"="cert-manager-webhook-ca" "secret_namespace"="cert-manager"
I1001 12:15:35.653455       1 server.go:146] cert-manager/webhook "msg"="listening for insecure healthz connections"  "address"=":6080"
I1001 12:15:35.653488       1 server.go:159] cert-manager/webhook "msg"="listening for secure connections"  "address"=":10250"
I1001 12:15:35.653509       1 server.go:185] cert-manager/webhook "msg"="registered pprof handlers"
I1001 12:15:35.654464       1 reflector.go:207] Starting reflector *v1.Secret (1m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
E1001 12:15:36.660296       1 dynamic_source.go:88] cert-manager/webhook "msg"="Failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input"  "interval"=1000000000
E1001 12:15:37.659501       1 dynamic_source.go:88] cert-manager/webhook "msg"="Failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input"  "interval"=1000000000
E1001 12:15:38.660871       1 dynamic_source.go:88] cert-manager/webhook "msg"="Failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input"  "interval"=1000000000
I1001 12:15:39.682232       1 dynamic_source.go:199] cert-manager/webhook "msg"="Updated serving TLS certificate"
I1001 12:15:45.080062       1 logs.go:58] http: TLS handshake error from 100.96.0.1:54726: remote error: tls: bad certificate
  • Multiple attempts to upgrade the management cluster result in the following output from tanzu CLI: 
    • Upgrading management cluster providers...
Error: failed to upgrade management cluster providers: failed to get providers upgrade information: failed to parse the management group: failed to find core provider from the current providers



Environment

1.x

Resolution

Workaround:

  1. On a machine with access to the tanzu CLI, run the tanzu login command to see the available management clusters and identify which one is the current login context for the CLI.
  2. To see the details of the management cluster, run tanzu management-cluster get
  3. Issue a command similar to the following to retrieve a kubeconfig for the management cluster
tanzu management-cluster kubeconfig get --admin --export-file MC-ADMIN-KUBECONFIG
  1. Issue a command similar to the following to set the context of kubectl to the management cluster:
kubectl config use-context my-mgmnt-cluster-admin@my-mgmnt-cluster
  1. Create a file called provider-definition.yaml with the following content:
apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Provider
metadata: 
  labels: 
    cluster.x-k8s.io/provider: cluster-api
    clusterctl.cluster.x-k8s.io: ""
    clusterctl.cluster.x-k8s.io/core: inventory
  name: cluster-api
  namespace: capi-system
providerName: cluster-api
type: CoreProvider
version: v0.3.14
  1. Run a command similar to the following:  
kubectl apply -f <path to provider-definition.yaml>

Additional Information

Powered by Wolken Software