Error: 403 Forbidden: The credentials were incorrect or the account specified has been locked - when performing a password update or remediate task for SDDC/NSX-T in SDDC Manager.
search cancel

Error: 403 Forbidden: The credentials were incorrect or the account specified has been locked - when performing a password update or remediate task for SDDC/NSX-T in SDDC Manager.

book

Article ID: 314647

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware NSX

Issue/Introduction

VMware Cloud Foundation (VCF) environments may encounter issues where the SDDC Manager is unable to perform password rotations or remediations for NSX-T Manager accounts. This occurs when credentials expire or multiple failed login attempts trigger an automated lockout policy within the NSX Manager nodes.

  • SDDC Manager Password Manager shows disconnected state for NSX admin, root, or audit accounts.
  • SDDC Manager UI displays: Failed to get NSX user details. Cause: The credentials were incorrect or the account specified has been locked.
  • The /var/log/vmware/vcf/operationsmanager/operationsmanager.log file shows the following trace:
    Caused by: org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Environment

  • VCF
  • VMware NSX

Cause

A synchronization failure or mismatch between credentials stored in the SDDC Manager database and the NSX appliance causes repeated authentication failures. These continuous failed attempts violate the NSX API security policy, leading to an automatic account lockout.

Resolution

To resolve this issue, unlock the admin account on the NSX-T Manager nodes, synchronize the credentials between NSX-T and SDDC Manager, and restore the default security policies.

Step 1: Unlock Account and Disable Lockout Policy

  1. Log in to each NSX-T Manager node via SSH using the admin credentials.

  2. Reset the admin account lockout counter:
    Note: Need to switch to root user using "st e" and use the root user password to run the following command.

    /usr/sbin/faillock --user admin --reset

  3. Temporarily disable the API lockout policy. This allows synchronization to occur without triggering further lockouts:
    Notes:
    1. Need to switch back to admin user with "su admin" to run the following commands.

    2. Need to run the below commands on only any one of the NSX nodes and those DB values will replicate to the other nodes. Replication should complete within a couple minutes.

      set auth-policy api lockout-period 0
      set auth-policy api lockout-reset-period 0
    3. On each NSX node run the following commands to confirm the status is now "disabled" for the two configurations.

      get auth-policy api lockout-period
      get auth-policy api lockout-reset-period

Step 2: Synchronize Passwords

Choose one of the following methods to synchronize the credentials.
Note: If the password listed in SDDC manager DB for the "admin" user works to log into the NSX nodes, there is no need to update (change) the password.

Method 1: Update the NSX-T password to match the SDDC Manager database

  1. Retrieve the current database password by running the lookup_passwords command on the SDDC Manager appliance.

  2. On the NSX-T Manager node, update the admin password to match the retrieved database password:
    passwd admin

Method 2: Update the SDDC Manager database to match a new password

Prerequisite: Take a snapshot of the SDDC Manager virtual machine before proceeding with this method.

  1. Log in to the SDDC Manager appliance as root via SSH.

  2. Generate an API token (replace <sso username> and <sso password> with your actual SSO administrator credentials):
    TOKEN=$(curl -d '{"username" : "<sso username>", "password" : "<sso password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')

  3. Query the system for the NSX-T API credentials to locate its ID:
    curl -X GET 'localhost/v1/system/credentials?entityType=NSXT_MANAGER&credentialType=API' -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN"

  4. Push the new password into the SDDC Manager database (replace <id from above step> and <new password> with the actual values):
    curl -X PUT 'localhost/v1/system/credentials/<id from above step>' -d '<new password>' -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN" | json_pp

Step 3: Finalize and Remediate

  1. Perform a rolling reboot of all NSX-T Manager nodes. This clears any active sessions and cached authentication states.

  2. Log in to the NSX Manager UI and verify that the NSX cluster is stable and healthy.

  3. Log in to the SDDC Manager UI and navigate to Security > Password Management.

  4. Run the REMEDIATE operation for the admin account. (Note: You can also run this for the root and audit accounts if necessary).

  5. Once remediation is successful, SSH back into any one of the NSX-T Manager nodes and restore the original lockout policy values.
    Note: The DB values will replicate to the other nodes. Replication should complete within a couple minutes.

    set auth-policy api lockout-period 900
    set auth-policy api lockout-reset-period 900

  6. On each NSX node run the following commands to confirm the status is now "disabled" for the two configurations.

    get auth-policy api lockout-period
    get auth-policy api lockout-reset-period

Additional Information

For more information on Authentication Policy Settings

Impact/Risks:

NOTE: The process involves minimal configuration changes on the NSX-T Managers.

  • There are no risks involved with these configuration changes.

  • This issue is being checked by Diagnostics for VMware Cloud Foundation.

  • The check is as follows:
    • Product: SDDC
    • Log File: /var/log/vmware/vcf/operationsmanager/operationsmanager.log
    • Log Expression Check "The credentials were incorrect or the account specified has been locked"
Powered by Wolken Software