Error: 403 Forbidden: The credentials were incorrect or the account specified has been locked - when performing a password update or remediate task for SDDC/NSX-T in SDDC Manager.

Products

• VMware Cloud Foundation
• VMware NSX

Issue/Introduction

VMware Cloud Foundation (VCF) environments may encounter issues where the SDDC Manager is unable to perform password rotations or remediations for NSX-T Manager accounts. This occurs when credentials expire or multiple failed login attempts trigger an automated lockout policy within the NSX Manager nodes.

• SDDC Manager Password Manager shows disconnected state for NSX admin, root, or audit accounts.
• SDDC Manager UI displays: Failed to get NSX user details. Cause: The credentials were incorrect or the account specified has been locked.
• The /var/log/vmware/vcf/operationsmanager/operationsmanager.log file shows the following trace:
Caused by: org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Environment

• VCF 5.x
• VCF 9.x
• VMware NSX

Cause

A synchronization failure or mismatch between credentials stored in the SDDC Manager database and the NSX appliance causes repeated authentication failures. These continuous failed attempts violate the NSX API security policy, leading to an automatic account lockout.

Resolution

To resolve this issue, unlock the admin account on the NSX-T Manager nodes, synchronize the credentials between NSX-T and SDDC Manager, and restore the default security policies.

Step 1: Unlock Account and Disable Lockout Policy

1. Log in to each NSX-T Manager node via SSH using the admin credentials.
   
   
2. Reset the admin account lockout counter:
   /usr/sbin/faillock --user admin --reset
   
   
3. Temporarily disable the API lockout policy. This allows synchronization to occur without triggering further lockouts:
   set auth-policy api lockout-period 0
set auth-policy api lockout-reset-period 0

Step 2: Synchronize Passwords

Choose one of the following methods to synchronize the credentials.

Method 1: Update the NSX-T password to match the SDDC Manager database

1. Retrieve the current database password by running the lookup_passwords command on the SDDC Manager appliance.
   
   
2. On the NSX-T Manager node, update the admin password to match the retrieved database password:
   passwd admin

Method 2: Update the SDDC Manager database to match a new password

Prerequisite: Take a snapshot of the SDDC Manager virtual machine before proceeding with this method.

1. Log in to the SDDC Manager appliance as root via SSH.
   
   
2. Generate an API token (replace <sso username> and <sso password> with your actual SSO administrator credentials):
   TOKEN=$(curl -d '{"username" : "<sso username>", "password" : "<sso password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
   
   
3. Query the system for the NSX-T API credentials to locate its ID:
   curl -X GET 'localhost/v1/system/credentials?entityType=NSXT_MANAGER&credentialType=API' -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN"
   
   
4. Push the new password into the SDDC Manager database (replace <id from above step> and <new password> with the actual values):
   curl -X PUT 'localhost/v1/system/credentials/<id from above step>' -d '<new password>' -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN" | json_pp

Step 3: Finalize and Remediate

1. Perform a rolling reboot of all NSX-T Manager nodes. This clears any active sessions and cached authentication states.
   
   
2. Log in to the NSX Manager UI and verify that the NSX cluster is stable and healthy.
   
   
3. Log in to the SDDC Manager UI and navigate to Security > Password Management.
   
   
4. Run the REMEDIATE operation for the admin account. (Note: You can also run this for the root and audit accounts if necessary).
   
   
5. Once remediation is successful, SSH back into each NSX-T Manager node and restore the original lockout policy values:
   set auth-policy api lockout-period 900
set auth-policy api lockout-reset-period 900

Additional Information

For more information on Authentication Policy Settings

Impact/Risks:

NOTE: The process involves minimal configuration changes on the NSX-T Managers.

• There are no risks involved with these configuration changes.
  
  
• This issue is being checked by Diagnostics for VMware Cloud Foundation.
  
  
• The check is as follows:
  • Product: SDDC
  • Log File: /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  • Log Expression Check "The credentials were incorrect or the account specified has been locked"