Retiring a TanzuKubernetesCluster (TKC) with Additional Trusted CA Certificates

Products

Tanzu Kubernetes Runtime VMware vSphere Kubernetes Service vSphere with Tanzu

Issue/Introduction

This KB article will advise on necessary steps to perform prior to retiring a TanzuKubernetesCluster (TKC) configured with Additional Trusted CA Certificates.

If the following steps are not performed before retiring the TKC, the system is unable to perform scaling or upgrade operations on the affected workload cluster.

While connected to the Supervisor cluster context:

If the TKC with Additional Trusted CA Certificates is retired before following this KB article, the below error will be present when describing the machines of nodes in the cluster:

kubectl get machines -n <cluster namespace>

kubectl describe machine -n <cluster namespace> <machine name>

- lastTransitionTime: "YYYY-MM-DDTHH:MM:SSZ"

    message: 'failed to resolve file source: secret not found: <NAMESPACE>/<CLUSTER_NAME>-user-trusted-ca-secret: secrets "<CLUSTER_NAME>-user-trusted-ca-secret" not found'

    reason: DataSecretGenerationFailed

    severity: Warning

    status: "False"

    type: BootstrapReady

Scaling and upgrade operations on the affected workload cluster are not working.

Environment

vSphere with Tanzu 8.0

TKG Service/VKS Service 3.3.0

Cause

When configuring a v1beta1 TanzuKubernetesCluster (TKC) with Additional Trusted CAs, a Kubernetes secret object is created containing the double base-64 encoded certificate values and the TKC is configured to point to the data within the secret object.

This secret object is named using the workload cluster's name and user-trusted-ca-secret:

<cluster-name>-user-trusted-ca-secret

Upon retiring a TKC, the above secret is deleted by the system.

Without this secret, the system is unable to perform scaling or upgrade operations on the affected workload cluster.

Existing applications and pods within the affected workload cluster will continue to work until the originally configured Additional Trusted CAs expire.

v1beta1 Cluster Example with Additional Trusted CAs

Resolution

Before retiring a Tanzu Kubernetes Cluster (TKC) where Additional Trusted CA Certificates are configured, the associated secret object must be transferred to the cluster object to avoid losing it.

If the TKC has not yet been retired, please see the Transfer the Trusted CA Secret section below to change the ownership of the AdditionalTrustedCA secret to the cluster object.

If the TKC was already retired before following the transfer steps, please see the Create a new Trusted CA Secret section below to manually create a secret containing the desired AdditionalTrustedCA configuration.

Transfer the Trusted CA Secret

Connect into the Supervisor cluster context

Retrieve the CLUSTER-UID of the intended workload cluster object:

kubectl get cluster -n <NAMESPACE> <CLUSTER-NAME> -o jsonpath={".metadata.uid"}

Locate the user-trusted-ca-secret for the intended workload cluster:

kubectl get secret -n <NAMESPACE> <CLUSTER-NAME>-user-trusted-ca-secret

Update the secret's cluster fields to transfer its ownership to the intended workload cluster object, where <CLUSTER-UID> is the uid found in Step 2:

kubectl patch secret -n <NAMESPACE> <CLUSTER-NAME>-user-trusted-ca-secret --type merge  -p '{"metadata": {"ownerReferences":[{"apiVersion":"cluster.x-k8s.io/v1beta1", "kind": "Cluster", "name":"<CLUSTER-NAME>", "uid": "<CLUSTER-UID>"}]}}'

Confirm that the secret was updated appropriately:

kubectl get secret -n <NAMESPACE> <CLUSTER_NAME>-user-trusted-ca-secret -o yaml | grep ownerReference -A 4

Create a new Trusted CA Secret

Connect into the Supervisor cluster context
Follow the steps under "Procedure: Existing Cluster" in v1beta1 Cluster Example with Additional Trusted CAs
Refer to the Troubleshooting Additional Trusted CA Errors for common issues configuring the Additional Trusted CAs

Additional Information

Documentation: Retiring TanzuKubernetesCluster Resources