SDDC Cluster Creation Fails at "Create Transport Node Profile" with Unauthorized {error_message=The credentials were incorrect or the account specified has been locked}

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

When attempting to create a new cluster in VMware Cloud Foundation (VCF), the workflow fails at the "Create NSX Transport Node Collection" stage.
The SDDC Manager UI displays the following error message:

A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.

Message: A problem has occurred on the server. Please retry or contact the
service provider and provide the reference token.
Remediation Message:
Reference Token: ######
Cause: Unauthorized (com.vmware.vapi.std.errors.unauthorized) => {
messages = [], data = struct => {error_message=The credentials were
incorrect or the account specified has been locked., error_code=403,
module_name=common-services), errorType = UNAUTHORIZED }

In the /var/log/vmware/vcf/domainmanager/domainmanager.log file on the SDDC Manager, you see errors Unauthorized (403) exception

DEBUG [vcf_dm,69..] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-2]  Contract task Create NSX Transport Node Collection input: {"transportNodeProfileId":"1c62####-####-####-####-########cb1e","nsxtManager":{"address":"nsxmanager.example.com","port":0,"username":"admin","password":"*****"},"wldDmClusterName"
DEBUG [vcf_dm,69..] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-2]  Created ApiClient connection to: nsxmanager.example.com

ERROR [vcf_dm,69..] [c.v.v.c.n.s.c.c.ComplexHelpers,dm-exec-2]  Exception occurred during NSX API invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized) (statusCode:403) => {
messages = [],
data =  => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},
errorType = UNAUTHORIZED

Environment

VMware Cloud Foundation (VCF) 5.x

VMware Cloud Foundation (VCF) 9.x

Cause

This issue occurs when the credentials for the NSX Manager admin account stored in the running workflow execution context differ from the actual credentials currently active in the SDDC Manager database or the NSX Manager itself.

When the workflow was restarted/retried, it continued using the cached (old) password from the original execution context, resulting in a 403 Unauthorized error

Resolution

 

1. SSH in to the SDDC Manager appliance using the vcf user account.
2. Enter the following command:
   

   lookup_passwords

3. Enter an entity type from the displayed list.
   For example: NSXT
4. When prompted, enter a user name and password for an SDDC Manager account assigned the ADMIN role.
5. Take note of the NSXT admin password that SDDC Manager has stored
6. Update the Workflow Parameter
   Once you have the correct password, update the running workflow to use these new credentials. Refer KB Re-try an existing workflow by modifying the workflow spec file
7. Restart the Failed Cluster create task from the SDDC Manager UI.