SDDC Manager unable to perform any password operations on NSX-T Managers, with the error: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Products

• VMware Cloud Foundation
• VMware NSX

Issue/Introduction

• NSX-T Manager credentials are expired - logging in with admin to the NSX-T Managers prompts a change of password
• SDDC Manager is unable to remediate credentials for the NSX-T Managers
• Any API calls made to the NSX-T Managers using the proper credentials fail from the SDDC Manager but work successfully from other sources.
• The API Calls from SDDC Manager fail with the following errors:{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Environment

• VMware Cloud Foundation
• VMware NSX
• VMware NSX-T Data Center

Cause

This is due to the password expiration on the admin account on the NSX-T Managers. As a result of the expired password, the password saved on SDDC Manager no longer works against the NSX-T Managers. Due to repeated failed login attempts via API, the NSX-T Managers lock out the SDDC Manager login attempts - even with the right credentials.

Resolution

This is a condition that may occur in a VMware NSX environment.

 

Workaround

1. Connect to each of the NSX-T Managers behind the NSX-T Load Balancer via SSH.
2. Login with admin credentials.
3. Run the following commands on each of the NSX-T Managers:
   • set auth-policy api lockout-period 0
   • set auth-policy api lockout-reset-period 0
4. Rolling reboot of NSX managers to release any locks on accounts.
   • Once all NSX managers are rebooted and the NSX cluster showed as stable proceed with the following steps.
5. Run the REMEDIATE password operation from the SDDC Manager UI against the admin account for NSX-T Manager - This time the operation should complete successfully.
6. Wait for a few minutes for the password to sync across all the NSX-T Manager nodes. 
7. Run the REMEDIATE password operation from the SDDC Manager UI against the root account for NSX-T Manager.
8. Restore the lockout-period and lockout-reset-period values back to the original value across all the NSX-T Managers, example:
   • set auth-policy api lockout-period 900
   • set auth-policy api lockout-reset-period 900

Alternatively, a rolling reboot of the NSX manager nodes could also unblock the admin/root user account and allow the passwords to be remediated from SDDC manager. 

Additional Information

Administration Guide 3.2

Impact/Risks:
NONE: The process involves minimal configuration changes on the NSX-T Managers.

There are no risks involved with these configuration changes.

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

• Product: SDDC
• Log File: /var/log/vmware/vcf/operationsmanager/operationsmanager.log
• Log Expression Check "The credentials were incorrect or the account specified has been locked"