Collecting Kubernetes Cluster Logs for tcakubernetesclusters in Telco Cloud Automation (TCA) 2.0

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

The purpose of this patch is to allow the collection of Kubernetes Cluster logs in TCA 2.0 using kubectl.

Symptoms:
In Telco Cloud Automation (TCA) 2.0, attempting to generate Kubernetes logs from the Administration > Troubleshooting > Technical Support Logs page may not show workload clusters in the cluster list. The support bundle generation is then unable to collect Kubernetes logs using the UI.

CCLI bundle generation also fails as CCLI is unable to list the workload clusters.

Environment

VMware Telco Cloud Automation 2.0

Cause

Collection of Kubernetes cluster logs for workload clusters created using cluster operator was not included as part of TCA 2.0.

Due to this, the previous procedure used prior to TCA 2.0 is not able to collect Kubernetes cluster logs either through the UI or the CCLI.

Resolution

Upgrade to Telco Cloud Automation 2.1 or higher where this issue is resolved.

Workaround:
The workaround is to follow the steps detailed here to collect Kubernetes cluster logs for tcakubernetesclusters in TCA 2.0.

Patch Kubernetes kbsctl

1. SSH into the TCA -CP as admin and switch user to root.
    systemctl stop bootstrapperd
2. Download the updated kbsctl binary attachment from this KB (see right panel).
3. Take a backup of the existing kbsctl in the /usr/bin path using the following command:
   bk /usr/bin/kbsctl /usr/bin/kbsctlBAK
4. Stop the bootstrapperd service using the following command:
   systemctl stop bootstrapperd
5. Replace the existing kbsctl binary in the /usr/bin path with the one downloaded in step 2.
6. Restart the bootstrapperd service using the following command:
   systemctl start bootstrapperd

Note: These steps should only be performed once.

Confirm patch

To verify the patch was properly applied, run the following command:
kbsctl techsupport --help
 
The updated kbsctl has 2 extra fields, ClusterOperatorFlag and MgmtClusterID, for collecting tcakubernetescluster k8s logs. Verify whether the fields are seen on stdout after running the above command.

Usage:
  kbsctl techsupport [flags]
 
Flags:
  -c, --ClusterOperatorFlag      Cluster Operator log bundle flag. 
  -m, --MgmtClusterID string     Management cluster ID for Tca Kubernetes Cluster log bundle
  -a, --apiserverd-addr string   the address of apiserverd (Example: 127.0.0.1:8888)
  -i, --cluster-ids string       collect logs for specified clusters only, seperated by commas
  -h, --help                     help for techsupport
  -o, --out string               the target directory of output (default "/root")
  -t, --timeout string           timeout to collect each cluster in minutes

Collecting Kubernetes Logs

Once the binary has been updated, the Kubernetes logs can be collected.

To collect tcakubernetescluster K8s logs you will need to specify -c as one of the parameters.

In the commands below, the -m flag accepts a management cluster id (generated by kbs). The management cluster id corresponds to the management cluster tcakubernetescluster is part of.

1. Obtain the management cluster ID generated by Kubernetes and for use with the log bundle by running the following command on the TCA-CP:
   kbsctl show managementclusters
2. Run the following to collect the Kubernetes log bundle for a single tcakubernetescluster. 
   kbsctl techsupport -c -i tcakubernetescluster_name -m mgmt_cluster_id
3. Run the following to collect the Kubernetes log bundle for multiple tcakubernetesclusters:
   kbsctl techsupport -c -i tcakubernetescluster_name1, tcakubernetescluster_name2, tcakubernetescluster_name3 -m mgmt_cluster_id
    

Once the command execution for log bundle collection completes, the TCA-CP stdout provides the path where the log bundle is stored. Typically, it is stored as /tmp/k8s-bootstrapper-TIMESTAMP.

Additional Information

Impact/Risks:

Only impacts Telco Cloud Automation 2.0. 

For all other versions of TCA, please refer to Collecting log bundles from Telco Cloud Automation (TCA) 2.x via CLI (90190)