Error: failed to create tko client: the server has asked for the client to provide credentials

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

When running the update-vc-tp.sh script (Update vCenter certificate for TKG cluster) on TCA-CP  to update a vCenter Server thumbprint, the job fails to complete.

• Workload clusters display the following error when checking their endpoints:
  Error in TLS server certificate thumbprint check: certificate thumbprints are not matched

• The update script status output later shows a Failed phase at 50% progress.

• Sub-job logs disclose the following error string:
  Reason: failed to create tko client: the server has asked for the client to provide credentials

• Checking the client certificate data from the k8s-bootstrapper-service endpoint confirms that the user client certificate is expired.

Environment

TCA: 3.2

TCP:5.0.1

Cause

• An expired client certificate within the bootstrapper configuration database causes the TKO client initialization to fail authentication checks against the cluster API server, blocking the execution of the thumbprint update meant for the workload cluster endpoints.

Resolution

1. Log in via SSH to the affected TCA-CP appliance as admin.

2. Obtain the bootstrapper service IP and management cluster UUID required for diagnostics:

   1. Run the following command to discover the Service IP:

      kubectl get svc -n tca-cp-cn k8s-bootstrapper-service
      

   2. Query the bootstrapper API locally to fetch the required management cluster UUID string:

      curl -s http://<k8s-bootstrapper-service-ip>:8888/api/v1/managementclusters | jq -r '.[] | "Cluster Name: \(.clusterName) -> UUID: \(.id)"'
      

3. Remediate the expired bootstrapper client certificate inside the database by applying the specific database workaround instructions detailed in KB Error: "failed with newly created client is not active" during Node Pool creation/customization.

4. Verify that the regenerated certificate is valid by pulling the configuration block and checking the output end date:

    

   curl -s http://<bootstrapper-service-ip>:8888/api/v1/managementcluster/<Mgmt_cluster_uuid>/kubeconfig -o /tmp/Mgmt_verify.kc.txt
   
   
   kubectl --kubeconfig /tmp/Mgmt_verify.kc.txt config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -enddate
   

   The <bootstrapper-service-ip> and <Mgmt_cluster_uuid> need to be replaced with values from Step 2.

5. Once verified, rerun the original vCenter thumbprint update script according to KB Update vCenter certificate for TKG cluster to resolve the unmatched certificate thumbprint error on the workload clusters.

 

Additional Information

 

• Update vCenter Certificate for TKG Cluster in TCA (345720)

• Error failed with newly created client in TCA (399339)