Error: "Cannot complete login due to an incorrect user name or password" when using guest customization in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• Deploying a VM with guest customization enabled intermittently fails with the following error:
  
  Cannot complete login due to an incorrect user name or password.
  
  
• Some VM deploys with guest customization enabled still complete without error.
• In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log on VMware Cloud Director (VCD), you see an error similar to:
  
  Caused by: java.io.IOException: Cannot complete login due to an incorrect user name or password.
          at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl$TransferWrapper.execute(NfcClientFacadeImpl.java:183)
          at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl.putFile(NfcClientFacadeImpl.java:240)
          at com.vmware.vcloud.val.filetransfer.impl.FileTransferToRemoteHostActivity$TransferToRemoteHostPhase.invokeCancelableOperation(FileTransferToRemoteHostActivity.java:112)
          ... 8 more
  Caused by: HTTP error on PUT https://<ESXi Host>.example.com/folder/<VM Name>/cust####################.cab?dsName=<datastore name>: 503
          at com.vmware.vcloud.val.nfc.client.impl.HttpTransferClientSession$StreamConsumer.execute(HttpTransferClientSession.java:110)
          at com.vmware.vcloud.val.nfc.client.impl.HttpTransferClientSession.putFile(HttpTransferClientSession.java:219)
          at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl$1.run(NfcClientFacadeImpl.java:244)
          at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl$1.run(NfcClientFacadeImpl.java:240)
          at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl$TransferWrapper.execute(NfcClientFacadeImpl.java:163)
          ... 10 more

Cause

ESXi Host has returned a 503 error when VCD was attempting to transfer the guest customization .cab file to the Host Datastore.

ESXi Host may not be in a healthy state.

Datastore may not be connected to ESXi Host.

Resolution

Ensure that all ESXi hosts in the destination cluster are healthy.

Ensure that all datastores are connected to the ESXi hosts.