Cluster creation fails due to incorrect vCenter credentials in TKGi

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

A customer is unable to create clusters and are seeing task errors such as:

Last Action Description:  Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: ########-####-####-####-############, broker-request-id: ########-####-####-####-############, task-id: ######, operation: create, error-message: 'master/########-####-####-####-############ (0)' is not running after update. Review logs for failed jobs: csi-controller, csi-syncer, csi-provisioner, csi-attacher, csi-resizer, ncp

Cause

Looking into the /var/vcap/sys/log/csi-controller/csi-controller.stderr.log file on the master node you can see errors that begin with:

{"level":"error","time":"<date>","caller":"<vsphere>/<virtualcenter>","msg":"failed to connect to VirtualCenter host: \"<vCenterHost>\". Err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.","TraceId": ...

(Error message shortened for brevity) 

This indicates an incorrect password.

Resolution

Verify the master node and bosh director are using the correct credentials. 

On master node: /var/vcap/jobs/csi-controller/config/csi-vsphere.conf
On Bosh director: /var/vcap/jobs/vsphere_cpi/config/cpi.json

If the credentials do not match what is currently being used it will be necessary to update the password in the TKGi tile UI.

In the OpsMan UI navigate to TKGi tile > Settings > Kubernetes Cloud Provider > Choose your Iaas > vSphere > update the vCenter Master Credentials