SDDC Manager unable to perform any password operations on NSX-T Managers, with the error: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Products

• VMware Cloud Foundation
• VMware NSX

Issue/Introduction

Reset the API lockout from the NSX-T Manager against the SDDC Manager VM, and thereby restore SDDC Manager's ability to manage NSX-T Managers and their credentials

Symptoms:

NSX-T Manager credentials are expired - logging in with admin to the NSX-T Managers prompts a change of password
 

SDDC Manager is unable to remediate credentials for the NSX-T Managers
 

Any API calls made to the NSX-T Managers using the proper credentials fail from the SDDC Manager, but work successfully from other sources.
 

The API Calls from SDDC Manager fail with the following errors:

{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

 

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

This is due to the password expiration on the admin account on the NSX-T Managers. As a result of the expired password, the password saved on SDDC Manager no longer works against the NSX-T Managers. Due to repeated failed login attempts via API, the NSX-T Managers lock out the SDDC Manager login attempts - even with the right credentials.

Resolution

1. Connect to each of the NSX-T Managers behind the NSX-T Load Balancer via SSH.

2. Login with admin credentials.

3. Run the following commands on each of the NSX-T Managers:

set auth-policy api lockout-period 0
set auth-policy api lockout-reset-period 0

4. Run the REMEDIATE password operation from the SDDC Manager UI against the admin account for NSX-T Manager - This time the operation should complete successfully.
Wait 2 minutes for the password to sync across all the NSX-T Manager nodes.

5. (Optional) Run the REMEDIATE password operation from the SDDC Manager UI against the root account for NSX-T Manager.

6. Restore the lockout-period and lockout-reset-period values back to the original value across all the NSX-T Managers:

set auth-policy api lockout-period 900
set auth-policy api lockout-reset-period 900

 

Additional Information

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-DB31B304-66A5-4516-9E55-2712D12B4F27.html

Impact/Risks:
NONE: The process involves minimal configuration changes on the NSX-T Managers. There is no risk involved with these configuration changes.

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

• Product: SDDC
• Log File: operationsmanager.log
• Log Expression Check "The credentials were incorrect or the account specified has been locked"