SDDC Manager unable to perform any password operations on NSX-T Managers, with the error: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Products

• VMware Cloud Foundation
• VMware NSX

Issue/Introduction

VMware Cloud Foundation (VCF) environments may encounter issues where the SDDC Manager is unable to perform password rotations or remediations for NSX-T Manager accounts. This occurs when credentials expire or multiple failed login attempts trigger an automated lockout policy within the NSX Manager nodes.

• SDDC Manager Password Manager shows failures for admin, root, or audit accounts.
• SDDC Manager UI displays: Failed to get NSX user details. Cause: The credentials were incorrect or the account specified has been locked.
• Resource shows a "Disconnected" state in Password Management.
• Operationsmanager log file shows the below: 
  
  /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  
  Caused by: com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: The credentials were incorrect or the account specified has been locked.
        at com.vmware.vcf.passwordmanager.helper.NsxtApiUtil.getUserDetails(NsxtApiUtil.java:162)
        at com.vmware.vcf.passwordmanager.update.changers.NsxtManagerApiChanger.doTest(NsxtManagerApiChanger.ja
va:114)
        ... 9 common frames omitted
Caused by: org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: "

{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Environment

• VCF 5.x
• VCF 9.x
• VMware NSX

Cause

A credential mismatch between the SDDC Manager database and the NSX appliance triggers the NSX API security policy. Repeated failed authentication attempts by SDDC Manager result in a locked account.

Resolution

1. Connect to each NSX-T Manager node via SSH using admin credentials.
2. Reset the admin account lockout:
   • /usr/sbin/faillock --user admin --reset
3. Temporarily disable the API lockout policy to allow synchronization
   
   • set auth-policy api lockout-period 0
   • set auth-policy api lockout-reset-period 0

4. Update the NSX admin password to match the SDDC Manager database using passwd admin.
   Note: Retrieve the current DB password using the lookup_passwords command on SDDC Manager.
5. Perform a rolling reboot of all NSX Manager nodes to clear active sessions and cached authentication states.
6. Verify the NSX cluster is stable in the NSX Manager UI. 
7. In the SDDC Manager UI, navigate to Security > Password Management and run the REMEDIATE operation for the admin account.
8. Run the REMEDIATE operation for the root and audit accounts if necessary.
9. Restore the original lockout policy values on each NSX-T Manager:
   
   • set auth-policy api lockout-period 900
   • set auth-policy api lockout-reset-period 900

Additional Information

For more information on Authentication Policy Settings

Impact/Risks:

NOTE: The process involves minimal configuration changes on the NSX-T Managers.

• There are no risks involved with these configuration changes.
  
  
• This issue is being checked by Diagnostics for VMware Cloud Foundation.
  
  
• The check is as follows:
  • Product: SDDC
  • Log File: /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  • Log Expression Check "The credentials were incorrect or the account specified has been locked"