VKS Cluster Deployment: Resolving the "illegal base64 data at input byte" Error When Adding a Second Additional Trusted CA

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When a cluster deployment is attempted, describing the failed cluster reveals that it remains stuck in a provisioning state with the following error:

kubectl describe cluster -n <cluster-namespace> <cluster-name>

error computing the desired state of the Cluster topology: failed to apply patches: failed to generate patches for patch "default": failed to call extension handler "generate-patches.runtime-extension": got failure response, please check controller logs for errors

Reviewing the logs from the runtime-extension-controller-manager reveals that the certificate contains illegal base64 data:

kubectl logs -n <runtime-extension-namespace> runtime-extension-controller-manager-xxx-xxx

E0422 12:13:19.030556       1 handler.go:129] "error during patch generation" err="validation errors in control plane trust configuration: [osConfiguration.trust.additionalTrustedCAs[cert.pem].data: Invalid value: \"cert.pem\": error retrieving data from secret cluster-ca: failed to decode base64 data for key cert.pem: illegal base64 data at input byte 0]" cluster="cluster-namespace/cluster01" namespace="cluster-namespace" name="cluster01" clusterclass="builtin-generic-v3.6.0" version="v1.35.2+vmware.1"

 

Environment

VKS 3.x

v1beta1/v1beta2 Cluster Class

Cause

This failure occurs when the double-encoded certificate contains an invalid character. This certificate is present on the supervisor cluster as per the documentation. In this instance, the certificate included an additional whitespace at the end of the text. You can identify this by decoding the string twice:

echo "<encoded-string>" | base64 -d | base64 -d

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
       <-- Empty whitespace was present in the decoded certificate

For the certificate to be accepted, there must be no additional characters or trailing whitespaces anywhere within the certificate text.

Resolution

To resolve this issue, first decode the double-encoded base64 string to inspect its contents:

echo "<encoded-string>" | base64 -d | base64 -d

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Ensure that the output contains no irregularities, hidden characters, or trailing whitespaces. Once the certificate is verified, save the proper certificate to a file and re-double-encode it using base64:

base64 -w 0 ca.crt | base64 -w 0

Next, update the secret from which the cluster retrieves the certificate using the corrected value. You can view the secret with the following command:

kubectl get secret -n <secret-namespace> <additional-ca-secret-name>

#additional-ca-1.yaml
apiVersion: v1
data:
  additional-ca-1: TFMwdExTMUNSGlSzZ3Jaa...VVNVWkpRMEMwdExTMHRDZz09 
  additional-ca-2: TFMwdExTMUNSGlSzZ3Jaa...VVNVsdhddhsjskkkjExTMHsDZz09 <-- Update here
kind: Secret
metadata:
  name: cluster01-user-trusted-ca-secret
  namespace: cluster-ns
type: Opaque

Once the the value is updated, re-deploy the cluster and it will not experience the issue. 

Additional Information

v1beta1/v1beta2 Example: Cluster with Additional Trusted CA Certificates for SSL/TLS