SDDC Cluster Creation Fails at "Create Transport Node Collection" with Invalid Parameter {0} and 403 Unauthorized
search cancel

SDDC Cluster Creation Fails at "Create Transport Node Collection" with Invalid Parameter {0} and 403 Unauthorized

book

Article ID: 424530

calendar_today

Updated On:

Products

VMware SDDC Manager VMware Cloud Foundation VCF Operations VMware NSX

Issue/Introduction

  • When attempting to create a new cluster in VMware Cloud Foundation (VCF), the workflow fails at the "Create NSX Transport Node Collection" stage.
  • The SDDC Manager UI displays the following error message 
    One or more transport node(s) realization failed while creating the transport node collection with profile [Profile-ID] on compute collection [Collection-ID] through NSX Manager [nsxmanager.example.com].

Invalid parameter: {0}

     

  • In the /var/log/vmware/vcf/domainmanager/domainmanager.log file on the SDDC Manager, you see errors indicating TRANSPORT_NODE_RELIZE_FAILED followed by an Unauthorized (403) exception 
    ERROR [vcf_dm,69...] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-29]  [#####] TRANSPORT_NODE_RELIZE_FAILED One or more transport node(s) realization failed while creating the transport node collection with profile 1c62####-####-####-####-########cb1e on compute collection eeb3####-####-####-####-########7166:domain-c##3 through NSX  Manager nsxmanager.example.com.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: One or more transport node(s) realization failed while creating the transport node collection with profile 1c62####-####-####-####-########cb1e on compute collection eeb3####-####-####-####-########7166:domain-c##3 through NSX Manager nsxmanager.example.com.
     
Caused by: java.lang.RuntimeException: Failed to realize transport node. Please refer logs.

     

    DEBUG [vcf_dm,69..] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-2]  Contract task Create NSX Transport Node Collection input: {"transportNodeProfileId":"1c62####-####-####-####-########cb1e","nsxtManager":{"address":"nsxmanager.example.com","port":0,"username":"svc-sddc-nsx","password":"*****"},"wldDmClusterName":"example-cluster","vcenter":{"address":"vcenter.example.com","port":0,"username":"[email protected]","password":"*****"},"wldClusterMoid":"domain-c##3","clusterImage":"e7a4####-####-####-####-########9d65"}
DEBUG [vcf_dm,69..] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,dm-exec-2]  Security config retrieved {"fipsMode":true}

DEBUG [vcf_dm,69..] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-2]  Creating ApiClient to https://nsxmanager.example.com:443 with username svc-sddc-nsx

DEBUG [vcf_dm,69..] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-2]  Created ApiClient connection to: nsxmanager.example.com

ERROR [vcf_dm,69..] [c.v.v.c.n.s.c.c.ComplexHelpers,dm-exec-2]  Exception occurred during NSX API invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized) (statusCode:403) => {
messages = [],
data =  => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},
errorType = UNAUTHORIZED
}

     

    ERROR [vcf_dm,69...] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-31]  [R6EGGD] VCF_ERRORS_GENERIC_INPUT_PARAM_ERROR Invalid parameter: {0}
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Invalid parameter: {0}
        at com.vmware.vcf.common.fsm.plugins.nsxt.policy.action.TransportNodeCollectionResolver.preCheck(TransportNodeCollectionResolver.java:343)
 
 
Caused by: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized) (statusCode:403) => {
    messages = [],
    data =  => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},
    errorType = UNAUTHORIZED

     

  • Additionally, the /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log file may show a PUT request to /credentials/ indicating a password update occurred shortly before the failure 
    INFO  [common,69...] [c.v.e.s.c.r.a.c.v.SecureCredentialController...] Received call to PUT /credentials/b92a####-####-####-####-########e7df
INFO  [common,69...] [c.v.v.l.a.a.ActivityLoggingInterceptor...] {"username":"vcfsvcs", ... "api":"/v1/system/credentials/b92a####-####-####-####-########e7df","httpMethod":"PUT","httpStatus":200,"operation":"Update an existing Credential in database", ...}

Environment

  • VMware Cloud Foundation (VCF) 9.x
  • NSX-T Data Center

Cause

This issue occurs when the credentials for the NSX Manager service account stored in the running workflow execution context differ from the actual credentials currently active in the SDDC Manager database or the NSX Manager itself.

This desynchronization often happens if:

  • The workflow initially failed (e.g., due to ESXi VIB installation issues).
  • During the troubleshooting of that initial failure, the NSX Manager password was rotated or updated (either manually or via password management operations).
  • When the workflow was restarted/retried, it continued using the cached (old) password from the original execution context, resulting in a 403 Unauthorized error and the generic Invalid parameter: {0}.

Resolution

To resolve this issue, retrieve the valid password currently stored in the SDDC Manager database and inject it into the failed workflow's parameters.

 

Steps to follow:

  1. Identify the NSX service account referenced in the failure logs, then retrieve its current password
    DEBUG [vcf_dm,69..] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-2]  Contract task Create NSX Transport Node Collection input: {"transportNodeProfileId":"1c62####-####-####-####-########cb1e","nsxtManager":{"address":"nsxmanager.example.com","port":0,"username":"svc-sddc-nsx","password":"*****"},"wldDmClusterName":"example-cluster","vcenter":{"address":"vcenter.example.com","port":0,"username":"[email protected]","password":"*****"},"wldClusterMoid":"domain-c##3","clusterImage":"e7a4####-####-####-####-########9d65"}

     

    1. SSH into the SDDC Manager with vcf user and su to root
    2. Create a token on the SDDC Manager. (You will need to update the username and password value) 
      TOKEN=$(curl -d '{"username" : "<sso_username>", "password" : "<sso_password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
    3. Pull the credentials from SDDC. 
      curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_pp | less


      Refer KB: Retrieve the service accounts credentials from SDDC Manager

  2. Update the Workflow Parameter
    Once you have the correct password, update the running workflow to use these new credentials. Refer KB Re-try an existing workflow by modifying the workflow spec file
  3. Restart the Failed Cluster create task from the SDDC Manager UI.
Powered by Wolken Software