Hardening cloud-provider-vsphere credential in TKGm workload clusters

Products

• VMware Tanzu Kubernetes Grid
• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

• To strictly adhere to the principle of least privilege, administrators may wish to scope credentials to match the specific functional requirements of the CPI.
• How to decouple the lifecycle management credentials from the workload cluster, ensuring that tenant processes interact only with the minimum required permissions.

Environment

TKG 2.5.x, 2.5.1, 2.5.2, 2.5.3, 2.5.4

TCA 3.x, 3.1, 3.1.1, 3.2, 3.4

TCP 5.0, 5.0.1, 5.1

Cause

• In TCA managed TKG deployments, the cloud-provider-vsphere-credentials Secret is located in the kube-system namespace of every workload cluster.
• Current TKG/management cluster architecture design prevents providing unique credentials for the Cloud Provider Interface (CPI) on a per-cluster basis, all workload clusters utilize the same CPI secret credentials.
• Since there is no mechanism to configure a unique username and password for each workload cluster, the highly-privileged credentials used for management cluster creation are propagated for the CPI.
• These credentials possess read-write access necessary for cluster lifecycle management (LCM), which is not necessary for the CPI module on the workload cluster.

Resolution

Workaround

Update the CPI user credentials

1. Create the TKG-CPI-Role role in vCenter Server with the following privileges:

   • Cns

     • Searchable

   • Datastore

     • Browse datastore

   • Sessions

     • Message

     • Validate session

   • Storage views

     • View

   • Virtual Machine

     • Provisioning

       • Allow read-only disk access

       • Allow virtual machine download

2. Create a local cpi user with vsphere.local as the domain name, and set an appropriate password.

3. Assign Permissions to Objects 

   1. Go to the Hosts and Clusters view.

   2. Right-click the Datacenter where TKG cluster will be deployed.

   3. Select Permissions and click Add.

   4. Enter the [email protected] user.

   5. Select the TKG-CPI-Role role.

   6. Check the Propagate to children checkbox.

4. SSH into the management cluster Control Plane node using the capv user.

5. Download the attached update_cpi_vcenter_credential script

6. Apply executable permissions

   • chmod +x ./update_cpi_vcenter_credential

7. Prepare an input CSV file containing the workload cluster names and the new vCenter credentials (username and password);

   Note: Ensure all passwords are provided in base64-encoded format:

   • echo -n '<vcenter_user_password>' | base64

8. Generate a CSV file that includes the workload cluster name, username, and base64 encoded password:

   • echo '<workload_cluster_name>,<vcenter_user_name>,<base64_encoded_vcenter_user_password>' > clusters.csv

9. Run the update_cpi_vcenter_credential script with the parameters of csv files:

   • ./update_cpi_vcenter_credential {csv_file_path}

     Example:

     ./update_cpi_vcenter_credential /tmp/clusters.csv

Known Limitations

• When running a standard (legacy plan-based) cluster, certain cluster operations require the update_cpi_vcenter_credential script to be reapplied.
  Refer to the following chart:

Cluster Operation	Standard cluster
Add Control plane	Reapply script required
Delete Control plane	Reapply script required
Modify Control plane	Reapply script required
Add worker node	Not needed
Delete worker node	Not needed
Modify worker node	Not needed
Deploy NF	Not needed
Upgrade cluster	Reapply script required
Certificate renew immediately from UI	Reapply script required
Certificate renew auto rotation	Not needed

Note: Updating the credentials for a classy (ClusterClass-based) clusters is a one-time operation and will not require the script to be re-applied. 

Additional Information

• There is no visibility/monitoring on expiring CPI passwords. The administrator is solely responsible for rotating this password. If the password expires or is changed in vCenter without a corresponding update to the Management Cluster secret, the CPI will be unable to access vCenter. In this event, you will need to re-run the update_cpi_vcenter_credential script with the new password for the user.
• There are no UI/API methods to perform to change the CPI username/password.