Products

VMware Telco Cloud Automation

Issue/Introduction

Users notice that their K8S VIMs are Disconnected in TCA-M, or the status of the Workload Cluster is disconnected (indicated with a red dot) in TCA-CP Appliance Management portal (9443) or the Harbor is in initiated state.

The harbor connection status will be in Initiated state as shown in below snippet

Also users can face "HttpStatus : 401 : unauthorized" error while adding the addons(eg systemsettings)

TCA web.log will show below errors

2024-10-01 10:56:34.245 UTC [https-jsse-nio-127.0.0.1-8443-exec-5, Ent: HybridityAdmin, Usr: abc@example, TxId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] INFO c.v. vca.hybridity.util. NSPRestClient- Token last updated 14+ minutes ago, refreshing token ..
2024-10-01 10:56:34.316 UTC [https-jsse-nio-127.0.0.1-8443-exec-5, Ent: HybridityAdmin, Usr: abc@example, TxId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] WARN c
c.v.vca. hybridity. util. NSPRestClient- Login to cloud https://<TCACP FQDN> failed, with status 401:
Usr: abc@example, TxId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] ERROR com. vmware. hybridity.nfvm. VnfUtils- Failed to query the TCA-CP Api POST:/hybridity/api//repositories/query.
java. lang.SecurityException: Login to cloud https://<TCACP FQDN> failed, with status 401:
2024-10-01 10:56:40.296 UTC [https-jsse-nio-127.0.0.1-8443-exec-8, Ent: HybridityAdmin, Usr: abc@example, TxId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] WARN c.v.vca.hybridity.util.NSPRestClient- Login to cloud https://<TCACP FQDN> failed, with status 401:

Note: It's not always necessary that the TCA UI show disconnected when there is an issue with kubeconfig. Please check the logs to verify the authentication errors

Environment

VMware Telco Cloud Automation 2.3, 2.3.0.1

Cause

Starting from TCA 2.3, TCA supports automatic certificate renewal for both management cluster and v2 workload clusters.

In certain cases, the certificates are renewed, but these are not synced to the TCA appliances. In such scenarios, users would notice that the relevant K8S VIM is Disconnected in TCA-M, or the status of this Cluster might be disconnected (indicated with a red dot) in TCA-CP Appliance Management portal (9443).

In such cases, it is required to manually update the Cluster certificates and / or kubeconfig stored in TCA database.

There are 2 parts to updating the CaaS Cluster certificates:

Updating the Cluster certificate within the Cluster itself
Updating the references of new Cluster Certificates within TCA-M and TCA-CP

This KB talks about step 2.

The assumption here is that the individual Cluster certificates have all been replaced correctly. If not, please follow the entire KB here: Manually renew cluster certificates

Resolution

This has been resolved in VMware Telco Cloud Automation 2.3.0.2 and newer versions (3.x).

For a manual resolution for VMware Telco Cloud Automation 2.3 or 2.3.0.1, please follow the steps below.

Synchronise the updated kubeconfig within TCA-M and TCA-CP

Note: All (upgraded and non-upgraded) Clusters require the kubeconfig to be synchronised

i.e. both upgraded and non-upgraded Clusters should follow this procedure.

First check the status of the Management Cluster within TCA-CP Appliance Management portal (9443). If this is red, then execute the steps below (including point 7) for the Management Cluster first, and then continue with the steps 2 - 7 for the Workload Clusters.
POST the following API call, from any machine that has access to the TCA-M web layer, to generate an authentication token:
```
curl -D - --location --insecure --request POST 'https://tca-m-url/hybridity/api/sessions' --header 'Accept: application/json' --header 'Content-Type: text/plain' --data-raw '{"username": "username","password": "plain_text_password"}'
```
Note: Replace tca-m-url and username and plain_text_password with the actual values in the command provided.
Take note of the x-hm-authorization from the output of the previous step:

Sample: 95XXXXX4:dXX2:4XX3:bXX2:7XXXXXXXXXX5
Update the TCA-M and TCA-CP database by synchronising the kubeconfig:
```
curl --location --insecure --request POST 'https://tca-m-fqdn/telco/api/caas/v2/clusters/cluster_name/syncKubeconfig' --header 'Accept: application/json' --header 'Content-Type: application/json' --header 'x-hm-authorization: auth-token'
```
Note: Replace tca-m-fqdn and cluster_name and auth-token with the actual values in the command provided.
Note: The operation can take several minutes, make a note of job_id received in the response which requires in the next step.
To ensure that the operation is succeeded, run the following API call:
```
curl --location --insecure --request GET 'https://tca-m-fqdn/hybridity/api/jobs/job_id_from_above_response' --header 'Accept: application/json' --header 'x-hm-authorization: auth-token'
```
Note: Replace tca-m-fqdn and auth-token with the actual values in the command provided.
Note: Take note of the isDone and didFail flags. The isDone flag should return true and the didFail flag should return false.
SSH login to TCA-CP to restart the services:
```
ssh admin@tca-cp
su -
```
Note: tca-cp where the cluster is configured
Restart the following TCA-CP services:
```
systemctl restart app-engine
systemctl restart web-engine
```
Note: In case of multiple TCA-CPs (i.e one for Mgmt cluster & one for Workload cluster) the app & web services should be started from both.

How to manually synchronize renewed CaaS kubeconfig for VMware Telco Cloud Automation 2.3

Article ID: 371488