SDDC Manager Fails to Load Cluster Network Information Due to NSX Password Mismatch
search cancel

SDDC Manager Fails to Load Cluster Network Information Due to NSX Password Mismatch

book

Article ID: 454557

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer VMware NSX

Issue/Introduction

  • VMware Cloud Foundation (VCF) SDDC Manager fails to load or query cluster network information, and related operations, such as adding or removing a cluster host, fail to execute.

    • Error Log Snippet:

      YYYY-MM-DDTHH:MM:SS.SSS+0000 ERROR [vcf_om,################################,####] [c.v.v.c.n.s.c.c.NsxtConnectionFactory,pool-#-thread-#] Unable to connect to <REDACTED_HOSTNAME>
com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized) (statusCode:403) => {
    messages = [],
    data =  => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},
    errorType = UNAUTHORIZED,
    challenge = <null>
}
  • The NSX Manager service account status reflects as normal (Active).
  • However, attempting to log into the NSX Manager UI using the service account secret from the SDDC Manager database fails.
  • These identical symptoms persist even after initiating a rotate password operation for the NSX Manager service account.

Environment

SDDC Manager 9.X

VMware NSX-T 9.X

Cause

The issue is caused by a password mismatch between the NSX service account and the SDDC Manager password database.

 

Resolution

  1. Log into the NSX Manager using the admin account to clear the account lockout. This can be done via one of the two options below:

    Option A: Clear Lockout via SSH

     

    • Log into each NSX-T Manager node via SSH using the admin credentials.

       

    • Reset the admin account lockout counter: /usr/sbin/faillock --user <SERVICE_ACCOUNT> --reset

    • Temporarily disable the API lockout policy to allow synchronization without triggering further lockouts: set auth-policy api lockout-period 0 set auth-policy api lockout-reset-period 0

    • Once the password rotation is successful, SSH back into each NSX-T Manager node and restore the original lockout policy values: set auth-policy api lockout-period 900 set auth-policy api lockout-reset-period 900

    Option B: Register the SDDC Manager IP to the Lockout Whitelist

    • Perform a snapshot of the SDDC Manager before proceeding.

    • Exempt the SDDC Manager from the NSX lockout policy by executing the following API command on the NSX Manager: curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' | jq ".lockout_immune_addresses += [ \"$(hostname -i)\" ]" | curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' -X PUT -H 'Content-Type:application/json' -d @-

    • After the rotate password operation is complete, remove the SDDC Manager IP from the lockout immunity list by executing: curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' | jq ".lockout_immune_addresses -= [ \"$(hostname -i)\" ]" | curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' -X PUT -H 'Content-Type:application/json' -d @-

  2. Execute the ROTATE PASSWORD operation for the NSX Manager service account using the SDDC Manager password workflow.

     

  3. Query and verify the password within the SDDC Manager database:

     

    • SSH into the SDDC Manager with the vcf user and use su to elevate to root.

       

    • Create a token on the SDDC Manager (Replace <sso_username> with [email protected] or the admin@local account and provide the respective password in <sso_password>): TOKEN=$(curl -d '{"username" : "<sso_username>", "password" : "<sso_password>"}' -H "Content-Type: application/json" -X POST http://<REDACTED_IP>/v1/tokens | jq -r '.accessToken')

    • Pull the credential details for all service accounts from the SDDC Manager using the token: curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_pp | less

    • Sample output

      {
      "serviceType" : "SDDC_MANAGER",
      "entityId" : "9189####-####-####-####-########8251",
      "username" : "svc-vcf-####-##",
      "id" : "688d####-####-####-####-########ad5e",
      "entityType" : "ESXI",
      "targetType" : "ESXI",
      "serviceId" : "2f7a####-####-####-####-########cb79",
      "secret" : "##############",
      "credentialType" : "SSH",

    • Locate the <REDACTED_SECRET> value in the output for the desired service account (svc-<####>-<####>) utilizing the serviceType and entityType (e.g., ESXI or NSX).

  4. Test logging into the NSX Manager UI with the service account credentials.

  5. Test the retrieval of Cluster Network information in SDDC Manager to confirm full functionality is restored.

Powered by Wolken Software