ERROR: "CNF135000: Kubernetes cluster unreachable: the server has asked for the client to provide credentials"

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Failed Helm deployments display error:
  CNF135000: Kubernetes cluster unreachable: the server has asked for the client to provide credentials
• In TCA Appliance Manager > Kubernetes Cluster status is in red.

Environment

2.3, 3.0

Cause

Kubeconfig is not synchronized and needs to be manually synchronized

Resolution

Synchronize the kubeconfig for the TCA-Manager (TCA-M) and TCA-CP

Note: All (upgraded and non-upgraded) Clusters require the kubeconfig to be synchronized.

The steps 1 through 4 should only be applied to TCA-M and not TCA-CP.

1. Generate an authentication token

   • In TCA 2.3, you can POST the following API call from any machine (TCA-M, TCA-CP, or any other Linux VM) that has access to the TCA-Manager web layer. 
   • In TCA 3.x, this API call cannot be run from the TCA-Manager itself. It must be executed from a seperate Linux VM that has access to the TCA-Manager.

   curl -D - --location --insecure --request POST 'https://tca-m-url/hybridity/api/sessions' --header 'Accept: application/json' --header 'Content-Type: text/plain' --data-raw '{"username": "username","password": "plain_text_password"}'

Note: Replace tca-m-url and username and plain_text_password with the actual values in the command provided. 

2. Take note of the x-hm-authorization UUID from the output of the previous step.
   
3. Update the TCA-M and TCA-CP database by synchronizing the kubeconfig:

curl --location --insecure --request POST 'https://tca-m-fqdn/telco/api/caas/v2/clusters/cluster_name/syncKubeconfig' --header 'Accept: application/json' --header 'Content-Type: application/json' --header 'x-hm-authorization: auth-token'

Note: Replace tca-m-fqdn and cluster_name and auth-token with the actual values in the command provided.
Note: The operation can take several minutes. 

4. To ensure that the operation is succeeded, run the following API call:

curl --location --insecure --request GET 'https://tca-m-fqdn/hybridity/api/jobs/job_id_from_above_response' --header 'Accept: application/json' --header 'x-hm-authorization: auth-token'

Note: Replace tca-m-fqdn, auth-token and job_id_from_above_response with the actual values in the command provided.
Note: Take note of the isDone and didFail flags in the json. The isDone flag should return true and the didFail flag should return false. 

5. SSH login to TCA-CP to restart the services:

ssh admin@tca-cp
su -

Note: tca-cp where the cluster is configured

6. Restart the following TCA-CP services:

systemctl restart app-engine
systemctl restart web-engine

Note: In case of multiple TCA-CPs (i.e one for Mgmt cluster & one for Workload cluster) the app & web services should be started from both.

Additional Information

If synchronizing kubeconfig does not resolve this issue, please review Manually renew cluster certificates to address additional certificate issues.