TKG management cluster not able to connect to vCenter after vCenter cert or thumbprint change
search cancel

TKG management cluster not able to connect to vCenter after vCenter cert or thumbprint change

book

Article ID: 337406

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

  • capv-controller-manager pod logs on the management cluster show similar errors to below:

    E1220 17:33:48.452774 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="unexpected error while probing vcenter for infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereCluster uploader-prod/uploader-autoscale: Post \"https://VCENTER-FQDN/sdk\": host \"VCENTER-FQDN:443\" thumbprint does not match \"<THUMBPRINT>\"" "controller"="vspherecluster" "name"="uploader-autoscale" "namespace"="uploader-prod"

  • You are unable to create or scale any workload clusters
  • If using auto-scaler the node will continue to attempt to provision and recreate based on settings in the cluster
  • If using Calico you will see errors similar: 

2025-04-02 22:29:09.930 [WARNING][85] felix/ipip_mgr.go 111: Failed to add IPIP tunnel device error=exit status 1

2025-04-02 22:29:09.930 [WARNING][85] felix/ipip_mgr.go 88: Failed configure IPIP tunnel device, retrying... error=exit status



Environment

VMware Tanzu Kubernetes Grid v1.5 to v2.x

Cause

This error occurs due to the vCenter certificate thumbprint changing and not being updated on the TKG management cluster objects and/or in the workload cluster object's metadata.

 

To confirm if there is a mismatch

 

In the Management Cluster context, update the {clustername}-vsphere-cpi-addon secret in the management cluster context


 -Get the actual secret name for your workload cluster
 kubectl get secret -A | grep cpi-addon


 - Save the data values information of the secret into a yaml file.  Make sure that the secret name here is correct and the same as the actual secret name as retrieved in the above command.
 kubectl get secret WC-vsphere-cpi-addon -o jsonpath={.data.values\\.yaml} | base64 -d > WC-vsphere-cpi-addon.yml


 - Open the yaml file and confirm that they match 


 -Confirm that the labels are there and match 
 kubectl get secret WC-vsphere-cpi-addon | grep label 
 ( this should look something like tkg.tanzu.vmware.com/cluster-name=WC and tkg.tanzu.vmware.com/addon-name=vsphere-cpi) 


 Verify in the Workload Cluster
 
 -The output of this command should show the thumbprint info
 kubectl -n tkg-system get secret vsphere-cpi-data-values -o jsonpath={.data.values\\.yaml} | base64 -d | grep -i thumbprint


 -The output of this command should show the thumbprint info as well. Compare these will doublecheck we are seeing the correct thumbprint in both locations
 kubectl -n kube-system get cm vsphere-cloud-config -o yaml

 

Resolution

This workaround is not valid for TKG stretch clusters deployed by Telco Cloud Automation. Please contact VMware support for assistance in updating TCA stretch cluster deployments.

 

For TKGm v2.3 and above

use the "tanzu mc credentials update" command to update the thumbprint in the Management Cluster and its Workload Clusters.  See the steps in Update Cluster Credentials for more info.

 

For TKGm v2.1 and v2.2

Update the workload cluster and management cluster data with the following steps. This should not impact the currently running nodes as this just updates the node metadata


Note: Please update the exported yaml file with the new value of thumbprint before replacing the secret. As a best practice verify if the secret is updated with the new thumbprint post replace.

 

To update the TLS thumbprint on each of the workload cluster:

In each of the commands, make sure to replace the string "WC" with your Workload Cluster name.

  1. In the Management Cluster context, update the {clustername}-vsphere-cpi-addon secret in the management cluster context

    # Get the actual secret name for your workload cluster
    kubectl get secret -A | grep cpi-addon

    # Save the data values information of the secret into a yaml file.  Make sure that the secret name here is correct and the same as the actual secret name as retrieved in the above command.
    kubectl get secret WC-vsphere-cpi-addon -o jsonpath={.data.values\\.yaml} | base64 -d > WC-vsphere-cpi-addon.yml

    # Open the yaml file in your favorite editor and change the thumbprint information.

    # Update the secret with the modified yaml file.
    kubectl create secret generic WC-vsphere-cpi-addon --type=tkg.tanzu.vmware.com/addon --from-file=values.yaml=WC-vsphere-cpi-addon.yml --dry-run=client -o yaml | kubectl replace -f -

    # Add labels to the secret
    kubectl label secret WC-vsphere-cpi-addon tkg.tanzu.vmware.com/cluster-name=WC
    kubectl label secret WC-vsphere-cpi-addon tkg.tanzu.vmware.com/addon-name=vsphere-cpi


  2. In the Workload Cluster context, verify that the secret vsphere-cpi-data-values in tkg-system namespace has been updated.  This should have been reconciled after the above secret has been updated.  

    # The output of this command should show the new thumbprint info
    kubectl -n tkg-system get secret vsphere-cpi-data-values -o jsonpath={.data.values\\.yaml} | base64 -d | grep -i thumbprint


  3. Verify the configmap is updated using the below command on the workload cluster context:

    # The output of this command should show the new thumbprint info
    kubectl -n kube-system get cm vsphere-cloud-config -o yaml


  4. Restart the vsphere-cloud-controller-manager pod so that the new configmap is mounted 

Note that he procedures above should be performed in each Workload Cluster.

 

To update the TLS thumbprint on the management cluster:

In each of the following commands, make sure to replace the string "MC" with your Management Cluster name.

  1. In the Management Cluster context, update the secret {management-clustername}-vsphere-cpi-data-values secret in tkg-system namespace

    # Get the actual secret name for your cluster
    kubectl -n tkg-system get secret | grep vsphere-cpi

    # Save the data values information of the secret into a yaml file.  Make sure that the secret name here is correct and the same as the actual secret name as retrieved in the above command
    kubectl -n tkg-system get secret MC-vsphere-cpi-data-values -o jsonpath={.data.values\\.yaml} | base64 -d > MC-vsphere-cpi-data-values.yml

    # Open the yaml file in your favorite editor and change the thumbprint information.

    # Update the secret with the modified yaml file.
    kubectl create secret generic MC-vsphere-cpi-data-values -n tkg-system --type=tkg.tanzu.vmware.com/addon --from-file=values.yaml=MC-vsphere-cpi-data-values.yml --dry-run=client -o yaml | kubectl replace -f -

    # Add labels to the secret
    kubectl label secret MC-vsphere-cpi-data-values -n tkg-system tkg.tanzu.vmware.com/cluster-name=MC
    kubectl label secret MC-vsphere-cpi-data-values -n tkg-system tkg.tanzu.vmware.com/addon-name=vsphere-cpi


Vsphere TLS Thumbprint also needs to be updated in the "vspherecluster" or "cluster", and "vspherevm" objects.  These has to be updated in all clusters.

 

  1. In the Management Cluster context, list all the vsphereclusters and clusters including the management cluster and note down their names as those will be needed in the next steps.

    kubectl get vsphereclusters -A
    kubectl get clusters -A


  2. For each of the clusters, edit the vspherecluster OR cluster object and update spec.thumbprint .

    If it's a legacy (non-classy) Workload Cluster, then edit the vspherecluster object and update the spec.thumbprint.

        kubectl edit vspherecluster WC

    Otherwise, if it's a classy Workload Cluster OR a Management Cluster, then edit the cluster object and update the spec.thumbprint.

        kubectl edit cluster WC


  3. Verify if the update is completed using the below command:

    kubectl get vspherecluster WC -o yaml

    OR

    kubectl get cluster WC -o yaml


  4. Restart the vsphere-cloud-controller-manager pod in the kube-system namespace in the Management Cluster.


  5. Scale down the CAPV deployment in the management cluster using the following command:


    kubectl scale deploy -n capv-system capv-controller-manager --replicas=0


  6. Update the webhook configurations in the management cluster to allow updates to the VSphereVM objects:

    kubectl patch validatingwebhookconfiguration capv-validating-webhook-configuration --patch '{"webhooks": [{"name": "validation.vspherevm.infrastructure.x-k8s.io", "failurePolicy": "Ignore"}]}'

    kubectl patch mutatingwebhookconfiguration capv-mutating-webhook-configuration --patch '{"webhooks": [{"name": "default.vspherevm.infrastructure.x-k8s.io", "failurePolicy": "Ignore"}]}'


  7. For each cluster, edit the VSphereVM objects with the updated thumbprint value with the following command:

    - Update the thumbprint on all the VSphereVM objects of the cluster <name-of-cluster> in the namespace <ns-of-cluster>

        kubectl get vspherevm -l cluster.x-k8s.io/cluster-name=<name-of-cluster> -n <ns-of-cluster> --no-headers=true | awk '{print $1}' | xargs kubectl patch vspherevm -n <ns-of-cluster> --type='merge' --patch '{"spec":{"thumbprint":"<new-thumbprint-value>"}}'


    - Confirm the updates on the VSphereVM objects by checking for the thumbprint update on the output of the VSphereVM objects by running the following commands:

            kubectl get vspherevm -l cluster.x-k8s.io/cluster-name=<name-of-cluster> -n <ns-of-cluster> -oyaml | grep thumbprint

    Note that you have to perform the above two commands in each cluster.


  8. Revert the changes to the webhook configurations in the management cluster by running the following commands:

    kubectl patch validatingwebhookconfiguration capv-validating-webhook-configuration --patch '{"webhooks": [{"name": "validation.vspherevm.infrastructure.x-k8s.io", "failurePolicy": "Fail"}]}'

    kubectl patch mutatingwebhookconfiguration capv-mutating-webhook-configuration --patch '{"webhooks": [{"name": "default.vspherevm.infrastructure.x-k8s.io", "failurePolicy": "Fail"}]}'


  9. Scale back up the CAPV deployment using the following command:

    kubectl scale deploy -n capv-system capv-controller-manager --replicas=1


 

 

For TKGm v1.5 and v1.6

Update the workload cluster and management cluster data with the following steps. This should not impact the currently running nodes as this just updates the node metadata


Note: Please update the exported yaml file with the new value of thumbprint before replacing the secret. As a best practice verify if the secret is updated with the new thumbprint post replace.

 

To update the TLS thumbprint on each of the workload cluster:

In each of the commands, make sure to replace the string "WC" with your Workload Cluster name.

  1. In the Management Cluster context, update the {clustername}-vsphere-cpi-addon secret in the management cluster context

    # Get the actual secret name for your workload cluster
    kubectl get secret -A | grep cpi-addon

    # Save the data values information of the secret into a yaml file.  Make sure that the secret name here is correct and the same as the actual secret name as retrieved in the above command.
    kubectl get secret WC-vsphere-cpi-addon -o jsonpath={.data.values\\.yaml} | base64 -d > WC-vsphere-cpi-addon.yml

    # Open the yaml file in your favorite editor and change the thumbprint information.

    # Update the secret with the modified yaml file.
    kubectl create secret generic WC-vsphere-cpi-addon --type=tkg.tanzu.vmware.com/addon --from-file=values.yaml=WC-vsphere-cpi-addon.yml --dry-run=client -o yaml | kubectl replace -f -

    # Add labels to the secret
    kubectl label secret WC-vsphere-cpi-addon tkg.tanzu.vmware.com/cluster-name=WC
    kubectl label secret WC-vsphere-cpi-addon tkg.tanzu.vmware.com/addon-name=vsphere-cpi


  2. In the Workload Cluster context, verify that the secret vsphere-cpi-data-values in tkg-system namespace has been updated.  This should have been reconciled after the above secret has been updated.  

    # The output of this command should show the new thumbprint info
    kubectl -n tkg-system get secret vsphere-cpi-data-values -o jsonpath={.data.values\\.yaml} | base64 -d | grep -i thumbprint


  3. Verify the configmap is updated using the below command on the workload cluster context:

    # The output of this command should show the new thumbprint info
    kubectl -n kube-system get cm vsphere-cloud-config -o yaml


  4. Restart the vsphere-cloud-controller-manager pod so that the new configmap is mounted 

Note that he procedures above should be performed in each Workload Cluster.

 

To update the TLS thumbprint on the management cluster:

In each of the following commands, make sure to replace the string "MC" with your Management Cluster name.

  1. In the Management Cluster context, update the {management-clustername}-vsphere-cpi-addon secret in the tkg-system namespace

    # Get the actual secret name for your Management Cluster
    kubectl -n tkg-system get secret | grep vsphere-cpi  

    # Save the data values information of the secret into a yaml file.  Make sure that the secret name here is correct and the same as the actual secret name as retrieved in the above command.
    kubectl -n tkg-system get secret MC-vsphere-cpi-addon -o jsonpath={.data.values\\.yaml} | base64 -d > MC-vsphere-cpi-addon.yml

    # Open the yaml file in your favorite editor and change the thumbprint information.  

    # Update the secret with the modified yaml file.
    kubectl create secret generic MC-vsphere-cpi-addon -n tkg-system --type=tkg.tanzu.vmware.com/addon --from-file=values.yaml=MC-vsphere-cpi-addon.yml --dry-run=client -o yaml | kubectl replace -f -

    # Add labels to the secret
    kubectl label secret MC-vsphere-cpi-addon -n tkg-system tkg.tanzu.vmware.com/cluster-name=MC
    kubectl label secret MC-vsphere-cpi-addon -n tkg-system tkg.tanzu.vmware.com/addon-name=vsphere-cpi

  2. Verify the configmap is updated using the below command:

    kubectl -n kube-system get cm vsphere-cloud-config -o yaml


  3. Restart the vsphere-cloud-controller-manager pod so that the new configmap is mounted 

 

Vsphere TLS Thumbprint also needs to be updated in the "vspherecluster" and "vspherevm" objects.  These has to be updated in all clusters.

  1. In the Management Cluster context, list all the vsphereclusters including the management cluster and note down their names as those will be needed in the next steps.

    kubectl get vsphereclusters -A
    NAMESPACE              NAME              AGE
    default                tkg-test          62d
    default                tkg-wld           83d
    tkg-system             tkg-mgmt          83d


  2. For each of the clusters, edit the vspherecluster CR and update spec.thumbprint. 

    kubectl edit vspherecluster WC
     

  3. Verify if the update is completed using the below command:

    kubectl get vspherecluster WC -o yaml


  4. Scale down the CAPV deployment in the management cluster context using the following command:

    kubectl scale deploy -n capv-system capv-controller-manager --replicas=0


  5. Update the CAPV validating webhook configuration in the management cluster to allow updates to the VSphereVM objects:

    kubectl patch validatingwebhookconfiguration capv-validating-webhook-configuration --patch '{"webhooks": [{"name": "validation.vspherevm.infrastructure.x-k8s.io", "failurePolicy": "Ignore"}]}'


  6. For each cluster, edit the VSphereVM objects with the updated thumbprint value with the following command:

    - Update the thumbprint on all the VSphereVM objects of the cluster <name-of-cluster> in the namespace <ns-of-cluster>

        kubectl get vspherevm -l cluster.x-k8s.io/cluster-name=<name-of-cluster> -n <ns-of-cluster> --no-headers=true | awk '{print $1}' | xargs kubectl patch vspherevm -n <ns-of-cluster> --type='merge' --patch '{"spec":{"thumbprint":"<new-thumbprint-value>"}}'


    - Confirm the updates on the VSphereVM objects by checking for the thumbprint update on the output of the VSphereVM objects by running the following commands:

            kubectl get vspherevm -l cluster.x-k8s.io/cluster-name=<name-of-cluster> -n <ns-of-cluster> -oyaml | grep thumbprint

    Note that you have to perform the above two commands in each cluster.

  7. Revert the changes to the webhook configuration in the management cluster by running the following command:

    kubectl patch validatingwebhookconfiguration capv-validating-webhook-configuration --patch '{"webhooks": [{"name": "validation.vspherevm.infrastructure.x-k8s.io", "failurePolicy": "Fail"}]}'


  8. Scale back up the CAPV deployment using the following command:

    kubectl scale deploy -n capv-system capv-controller-manager --replicas=1
Powered by Wolken Software