Management Cluster Upgrade Failed To Find Core Provider
book
Article ID: 317449
calendar_today
Updated On:
Products
VMware Tanzu Kubernetes Grid 1.x
Issue/Introduction
Symptoms:
While attempting to upgrade a Tanzu Kubernetes Grid (TKG) management cluster version from 1.3.1 to 1.4.0 users see similar to the following in the tanzu CLI output after the first upgrade attempt:
updating additional components: 'addons-management/tanzu-addons-manager' ...
updating additional components: 'tkr/tkr-controller' ...
updating additional components: 'addons-management/core-package-repo' ...
Error: failed to deploy additional components after kubernetes upgrade: error while upgrading additional component 'addons-management/standard-package-repo': kubectl apply failed, output: secret/mgmt-cluster-standard-package-repository-crs configured
Error from server (InternalError): error when creating "/tmp/kubeapply-211654953": Internal error occurred: failed calling webhook "default.clusterresourceset.addons.cluster.x-k8s.io": Post "https://capi-webhook-service.capi-webhook-system.svc:443/mutate-addons-cluster-x-k8s-io-v1alpha3-clusterresourceset?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
: exit status 1
The following messages are seen in the cert-manager-webhook pod logs:
W1001 12:15:35.653078 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1001 12:15:35.653294 1 webhook.go:57] cert-manager/webhook "msg"="using dynamic certificate generating using CA stored in Secret resource" "secret_name"="cert-manager-webhook-ca" "secret_namespace"="cert-manager"
I1001 12:15:35.653455 1 server.go:146] cert-manager/webhook "msg"="listening for insecure healthz connections" "address"=":6080"
I1001 12:15:35.653488 1 server.go:159] cert-manager/webhook "msg"="listening for secure connections" "address"=":10250"
I1001 12:15:35.653509 1 server.go:185] cert-manager/webhook "msg"="registered pprof handlers"
I1001 12:15:35.654464 1 reflector.go:207] Starting reflector *v1.Secret (1m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
E1001 12:15:36.660296 1 dynamic_source.go:88] cert-manager/webhook "msg"="Failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"=1000000000
E1001 12:15:37.659501 1 dynamic_source.go:88] cert-manager/webhook "msg"="Failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"=1000000000
E1001 12:15:38.660871 1 dynamic_source.go:88] cert-manager/webhook "msg"="Failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"=1000000000
I1001 12:15:39.682232 1 dynamic_source.go:199] cert-manager/webhook "msg"="Updated serving TLS certificate"
I1001 12:15:45.080062 1 logs.go:58] http: TLS handshake error from 100.96.0.1:54726: remote error: tls: bad certificate
Multiple attempts to upgrade the management cluster result in the following output from tanzu CLI:
Upgrading management cluster providers...
Error: failed to upgrade management cluster providers: failed to get providers upgrade information: failed to parse the management group: failed to find core provider from the current providers
Environment
1.x
Resolution
Workaround:
On a machine with access to the tanzu CLI, run the tanzu login command to see the available management clusters and identify which one is the current login context for the CLI.
To see the details of the management cluster, run tanzu management-cluster get
Issue a command similar to the following to retrieve a kubeconfigfor the management cluster
tanzu management-cluster kubeconfig get --admin --export-file MC-ADMIN-KUBECONFIG
Issue a command similar to the following to set the context of kubectl to the management cluster: