Patch Script for TKG Cluster Certificate Renewal

Products

• VMware Telco Cloud Automation

Issue/Introduction

This document contains the Telco Cloud Automation (TCA) scripts to renew TKG cluster certificates and update the certificate data and KUBECONFIG stored in TCA Manager and TCA-CP appliances.

Environment

2.0, 2.1.x, and 2.2

Resolution

1.  Validate jumpbox environment for prerequisites

The script is required to be executed from a linux machine which has access to TCA-M and all the TCA-CP involved. Please verify and confirm that required ports 22 (SSH), 443 (web UI) and 9443 (appliance management) are open from jumpbox to TCA-M & TCA-CP(s) involved in the cert renewal process. 

The jumpbox machine requires the following packages to be installed:

1. python3 (3.8.10 or higher)
2. paramiko (2.11.0 or higher)
3. requests (2.28.2 or higher)
4. boto3 (1.26.119 or higher)

Python 3.8.10
# For more details on installing python, review the Python Beginners Guide. 

# If paramiko package is NOT found then use following command to install 
python3 -m pip install paramiko

# If requests package is NOT found then use following command to install 
python3 -m pip install requests

# If boto3 package is NOT found then use following command to install 
python3 -m pip install boto3

2. Download patch scripts and untar them

Download the version specific patch script file from this article and save the file to the /tmp folder of the jumpbox machine. 
NOTE: Please empty the /tmp directory prior to copying over the patch script file, as other files present in /tmp can interfere with the patch file script. 

From the /tmp directory, untar the patch script files using the following commands:
# unzip the tarball to get the scripts and files under the cert-rotation-v1-X folder, where X can be the version of TCA.
  tar -xzvf cert-rotation-v1-X.tar.gz
# change to the cert-rotation-v1 directory
  cd cert-rotation-v1-X
# unzip the tarball to get the master scripts for cert-renewal 
  tar -xzvf master-kubeconfig-script-main.tar.gz 
# change to the master-kubeconfig-script-main directory
  cd master-kubeconfig-script-main

3. (Optional) Update the included config.json file with AWS Secret Manager details

Using a text editor, update the config.json file with AWS Secret Manager access tokens to fetch TCA & TCA-CP passwords from the password manager.

  vi config.json
{
  "SECRET_NAME": "<secret_key_name>",
  "REGION_NAME": "<region_name>",
  "AWS_ACCESS_KEY_ID":"<id>",
  "AWS_SECRET_ACCESS_KEY":"<key>",
  "AWS_SESSION_TOKEN":"<token>"
}

4. Run the retrieve_cluster_details.py script to get cert validity from TCA-M

Use chmod to update the execution permissions on the script, then run with the below command. This will generate a clusters.json file with all clusters certificate expiration details.

NOTE: A csv file with cluster cert expiry details (clusterData.csv) will also be created in the same path where the script is invoked. This can be used for reporting and planning purposes.

# change access permissions for retrieve_cluster_details.py file
  chmod +x retrieve_cluster_details.py 

# execute the script to retrieve cert validity details of clusters
  ./retrieve_cluster_details.py --tca_m_fqdn <tca_m_fqdn>

# ------ALTERNATE_OPTIONS------
# Only retrieve clusters showing kubeconfig expiring before the provided date
  ./retrieve_cluster_details.py --tca_m_fqdn <tca_m_fqdn> --date YYYY-MM-DD 

# Check the log file generated on on tca-manager appliance for errors.
  cat /tmp/cluster-kubeconfig-tool/run.log | grep ERROR

# NOTE: Verify the clusters.json file size. If the file is 0 bytes, re-run the script to ensure the file is correctly saved to the jumpbox.

5.  Create a backup of the TCA-M mongoDB collections

Log into TCA Manager as the admin user and use the following command to create backups of the following two collections:
mongodump --db hybridity --collection VimTenants --archive=/tmp/VimTenants.json
  mongodump --db hybridity --collection K8sClusterDetails --archive=/tmp/K8sClusterDetails.json

6. Run the cluster certificate rotation script from the jumpbox

Update the permissions for the update-kubeconfig.py file:
chmod +x update-kubeconfig.py

Once cluster and password details are finalized and a mongoDB backup has been created, trigger the master script to invoke the certificate rotation.

NOTE: The script expects config.json and clusters.json to be present in the same directory as update-kubeconfig.py.

# To specify certain clusters for certificate renewal, the --cert_renew_clusters flag should be used

# ./update-kubeconfig.py  --tca_m_fqdn <tca_m_fqdn> --cert_renew_clusters <cluster_name_1> <cluster_name2> 

NOTE: TCA 2.0 requires the --skip_appliance_mgmt_patch true flag to be used.

Example for TCA 2.0, 2.1.x, and 2.2:
# ./update-kubeconfig.py --tca_m_fqdn 10.105.45.75 --cert_renew_clusters wrkld_cluster wrkld_cluster_2

Example for TCA 2.0:
# ./update-kubeconfig.py --tca_m_fqdn 10.105.45.75 --cert_renew_clusters wrkld_cluster wrkld_cluster_2 --skip_appliance_mgmt_patch true

# ------ADDITIONAL_OPTIONS------

# If the script requires to be ran again on the same market TCA-CP, the TCA-CP war file patch can be skipped by using the --skip_appliance_mgmt_patch flag.
./update-kubeconfig.py  --tca_m_fqdn <tca_m_fqdn> --skip_appliance_mgmt_patch true

# Logging for the operation is located in kubeconfig-update.log.
vi kubeconfig-update.log

 

 

Additional Information

If the AWS Secret Manager details are unavailable, the alternate option is to manually create a creds.json file with the TCA-M and TCA-CP credentials.

Using a text editor, create a creds.json file and populate it with all TCA-M & TCA-CP passwords, using the below format for reference:

  vi creds.json

Include base64 encoded passwords for all the TCA-CPs involved in the the creds.json file. You can encode a password into Base64 by executing the following command:
echo -n 'Password' | base64
NOTE: The -n flag is required to prevent a line break from being added to the encoded password.


creds.json:

{
    "<tca_m_fqdn>": {
        "admin": "<base64 encoded password for tca m appliance for admin user>"
    },
    "<tca_cp_fqdn_1>": {
        "admin": "<base64 encoded password for tca cp appliance for admin user>",
        "root": "<base64 encoded password for tca cp appliance for root user>"
    },
    "<tca_cp_fqdn_2>": {
        "admin": "<base64 encoded password for tca cp appliance for admin user>",
        "root": "<base64 encoded password for tca cp appliance for root user>"
    }
 }

The script can then be executed by passing the creds.json file with the --tca_cp_creds flag:
./update-kubeconfig.py --tca_cp_creds creds.json --tca_m_fqdn tca_m_fqdn
NOTE: Replace tca_m_fqdn with the FQDN of the TCA Manager.