SDDC Manager unable to perform any password operations on NSX-T Managers, with the error: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Products

• VMware Cloud Foundation
• VMware NSX

Issue/Introduction

• NSX-T Manager credentials are expired - logging in with admin to the NSX-T Managers prompts a change of password
• SDDC Manager is unable to remediate credentials for the NSX-T Managers
• Any API calls made to the NSX-T Managers using the proper credentials fail from the SDDC Manager but work successfully from other sources.
• The API Calls from SDDC Manager fail with the following errors:{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Environment

• VMware Cloud Foundation
• VMware NSX
• VMware NSX-T Data Center

Cause

This is due to the password expiration on the admin account on the NSX-T Managers. As a result of the expired password, the password saved on SDDC Manager no longer works against the NSX-T Managers. Due to repeated failed login attempts via API, the NSX-T Managers lock out the SDDC Manager login attempts - even with the right credentials.

Resolution

This is a condition that may occur in a VMware NSX environment.

 

Workaround

1. Connect to each of the NSX-T Managers behind the NSX-T Load Balancer via SSH.
2. Login with admin credentials.
3. Run the following commands on each of the NSX-T Managers:
   • set auth-policy api lockout-period 0
   • set auth-policy api lockout-reset-period 0
4. Run the REMEDIATE password operation from the SDDC Manager UI against the admin account for NSX-T Manager - This time the operation should complete successfully.
5. Wait for a few minutes for the password to sync across all the NSX-T Manager nodes. 
6. Run the REMEDIATE password operation from the SDDC Manager UI against the root account for NSX-T Manager.
7. Restore the lockout-period and lockout-reset-period values back to the original value across all the NSX-T Managers, example:
   • set auth-policy api lockout-period 900
   • set auth-policy api lockout-reset-period 900

Alternatively, a rolling reboot of the NSX manager nodes could also unblock the admin/root user account and allow the passwords to be remediated from SDDC manager. 

Additional Information

Administration Guide 3.2

Impact/Risks:
NONE: The process involves minimal configuration changes on the NSX-T Managers.

There are no risks involved with these configuration changes.

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

• Product: SDDC
• Log File: /var/log/vmware/vcf/operationsmanager/operationsmanager.log
• Log Expression Check "The credentials were incorrect or the account specified has been locked"