SDDC Manager unable to perform any password operations on NSX-T Managers, with the error: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Products

• VMware Cloud Foundation
• VMware NSX

Issue/Introduction

• NSX-T Manager credentials are expired - logging in with admin to the NSX-T Managers prompts a change of password
   
• SDDC Manager is unable to remediate credentials for the NSX-T Managers
   
• Any API calls made to the NSX-T Managers using the proper credentials fail from the SDDC Manager but work successfully from other sources.
   
• The API Calls from SDDC Manager fail with the following errors:{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Environment

VMware NSX
VMware NSX-T Data Center

Cause

This is due to the password expiration on the admin account on the NSX-T Managers. As a result of the expired password, the password saved on SDDC Manager no longer works against the NSX-T Managers. Due to repeated failed login attempts via API, the NSX-T Managers lock out the SDDC Manager login attempts - even with the right credentials.

Resolution

This is a condition that may occur in a VMware NSX environment.

 

Workaround

1. Connect to each of the NSX-T Managers behind the NSX-T Load Balancer via SSH.

2. Login with admin credentials.

3. Run the following commands on each of the NSX-T Managers:

set auth-policy api lockout-period 0
set auth-policy api lockout-reset-period 0

4. Run the REMEDIATE password operation from the SDDC Manager UI against the admin account for NSX-T Manager - This time the operation should complete successfully.
Wait 2 minutes for the password to sync across all the NSX-T Manager nodes.

5. (Optional) Run the REMEDIATE password operation from the SDDC Manager UI against the root account for NSX-T Manager.

6. Restore the lockout-period and lockout-reset-period values back to the original value across all the NSX-T Managers:

set auth-policy api lockout-period 900
set auth-policy api lockout-reset-period 900

 

Additional Information

https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/3-2/administration-guide.html

Impact/Risks:
NONE: The process involves minimal configuration changes on the NSX-T Managers.

There are no risks involved with these configuration changes.

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

• Product: SDDC
• Log File: operationsmanager.log
• Log Expression Check "The credentials were incorrect or the account specified has been locked"