TKG cluster creation or upgrade fails with error - unable to configure variables for provider installation

Products

• VMware

Issue/Introduction

When decoded password value includes a single quote ( ' ) within the string value, issue might arise while unmarshaling the vSphere credentials stating an error as "Unable to get vSphere credentials from secret".

Symptoms:
You will fail to perform TKG CRUD operations and will run into the following error

tanzu management-cluster upgrade

Upgrading management cluster 'tkg-mgmt' to TKG version 'v1.4.0' with Kubernetes version 'v1.21.2+vmware.1'. Are you sure? [y/N]: y
cluster specific secret is not present, fallback on bootstrap credential secret
Error: unable to configure variables for provider installation: failed to configure
Vsphere credentials: unable to get vsphere credentials from secret: unable to retrieve vSphere credentials: yaml: line 1: did not find expected key

Environment

VMware Tanzu Kubernetes Grid 1.x

Cause

Currently the default yaml template has single quotes for username and password for vSphere related credentials as shown below

apiVersion: v1
kind: Secret
metadata:
 labels:
 cluster.x-k8s.io/provider: infrastructure-vsphere
 name: vSphere-credentials
 namespace: capv-system
stringData:
 credentials.yaml: |-
 username: '${VSPHERE_USERNAME}'
 password: '${VSPHERE_PASSWORD}'
type: Opaque

Since the password has single quote (eg: pass'word) the password supplied to the secret will be as

password : 'pass'word'

Resolution

A permanent resolution is targeted for a future version of TKG however a workaround is available.

Workaround:
You will need to update the credentials (there is no need to change the password, you can use the current password (eg: pass'word //without any quotes) as new password for credentials.

In order to update the credentials for management cluster including the workload cluster, please run the below set of commands from Tanzu CLI.

tanzu mc credentials update --cascading
? Enter vSphere username
? Enter vSphere password