Credential operations fail on NSX Components in SDDC Manager

Products

VMware Cloud Foundation

Issue/Introduction

This article helps troubleshoot failed credential operations with NSX-T Components. When attempting to rotate, update, or remediate a password for an NSX 4.x or NSX-T component in the SDDC Manager UI, you get error:

"Password management operation failed"

In the operationsmanager.log you see similar errors:

YYYY-MM-DDTHH:MM:SS.sssZ DEBUG [vcf_om,xxxxxxxxxxxxx,abcd] [c.v.v.p.helper.NsxtApiUtil,om-exec-5] Failed to get NSXT user details : {"module_name":"com
mon-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403} with status : 
YYYY-MM-DDTHH:MM:SS.sssZ ERROR [vcf_om,xxxxxxxxxxxxx,abcd] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-5] The credentials were incorrect or the account specified has been locked.
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: The credentials were incorrect or the account specified

Environment

VMware Cloud Foundation 5.x
VMware Cloud Foundation 4.0.x
VMware Cloud Foundation 3.0.x

Cause

This issue could be caused by the following:

NSX Manager passwords have expired.
NSX Manager passwords have been change manually outside of SDDC.

Resolution

Pull the most recent passwords from the SDDC Manager

lookup_passwords
API Explorer

Steps for the NSX-T Managers
Only needs to be performed on one manager per cluster.

Log in to the NSX-T manager as root. (Either from a console window or SSH)
Clear password history

echo "" >/etc/security/opasswd
Run the command

/etc/init.d/nsx-mp-api-server stop
Set the password(s) to match what is present in SDDC DB.

passwd
passwd admin
passwd audit
Run command: touch /var/vmware/nsx/reset_cluster_credentials
Run the command: /etc/init.d/nsx-mp-api-server start
Verify the accounts are not locked out with pam_tally2 or faillock

For VCF versions 4.3.0.0 up till 5.0.0.1, use:

pam_tally2 -u root -r
pam_tally2 -u admin -r
pam_tally2 -u audit -r

For VCF versions starting from VCF 5.1.0.0, use:

/usr/sbin/faillock --user root –reset
/usr/sbin/faillock --user admin –reset
/usr/sbin/faillock --user audit --reset

8. Retry the credential operation from the SDDC Manager UI.

Steps for the NSX-T Edges

Log in to the NSX-T edge as root. (Either from a console window or SSH)
Clear password history:

echo "" >/etc/security/opasswd
Run the command: /etc/init.d/nsx-edge-api-server stop
Set the password(s) to match what is present in SDDC DB.

passwd
passwd admin
passwd audit
Run the command: touch /var/vmware/nsx/reset_cluster_credentials
Run the command: /etc/init.d/nsx-edge-api-server start
Verify the accounts are not locked out with pam_tally2 or faillock

For VCF versions 4.3.0.0 up till 5.0.0.1, use:

pam_tally2 -u root -r
pam_tally2 -u admin -r
pam_tally2 -u audit -r

For VCF versions starting from VCF 5.1.0.0, use:

/usr/sbin/faillock --user root –reset
/usr/sbin/faillock --user admin –reset
/usr/sbin/faillock --user audit --reset

8. Retry the credential operation from the SDDC Manager UI.

Steps to change password expiration on NSX-T edges and Managers:

Connect to the NSX-T Manager or NSX-T Edge with the admin account.
You can elevate to admin from a root connection with su admin or st en.
Reset the expiration period.
1. You can clear the expiration as the admin user:
  
  clear user admin password-expiration
  clear user root password-expiration
  clear user audit password-expiration
2. Set the expiration period for between 1 and 9999 days.

nsxtmgr> set user admin password-expiration 9999
nsxtmgr> set user audit password-expiration 9999
nsxtmgr> set user root password-expiration 9999

Additional Information

Check to see if there's any locks:

curl http://localhost/locks | json_pp > releaseLock.json
curl -X PUT -H "Content-Type:application/json" http://localhost/locks -d @releaseLock.json

SDDC Manager unable to perform any password operations on NSX-T Managers, with the error: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}
If the password has expired and you're not able to reset it from the CLI/Console you'll have to reset it from the GRUB. See Resetting the Passwords of an Appliance.
SDDC Manager password operations are not allowed because of a failed password task
SDDC manager falsely shows the passwords of NSX-T components as expired during precheck/upgrade.

Log file /var/log/vmware/vcf/lcm/lcm-debug.log will show following entries :

2021-04-09T13:41:44.830+0000 INFO  [vcf_lcm,0000000000000000,0000,precheckId=########-####-####-####-25e8fb993243,resourceType=NSX_T,resourceId=nsx.test.local] [c.v.e.s.l.p.i.nsxt.NsxtPrimitiveImpl,pool-3-thread-49] Completed precheck task NSX_T_PASSWORD_VALIDITY_CHECK  on resource id nsx.test.local with status RED

- Executing the command "get user admin password-expiration" on NSX-T Manager may show as "Password expiration not configured for this user".
  - Cause: Password expiry is disabled and/or is set to 'never expire' (admin) or '-1' (root) for the NSX-T component, the precheck will falsely show the password as expired or the schedule rotation may fail.
  - Workaround: Set the password expiry to 9999 or a lesser number by using the command "set user admin password-expiration 9999" (as admin) OR 'chage -M 9999 root' (as root) before performing precheck/upgrade or remediation/rotation.
    - If remediation fails, set the lockout to 0 by following KB: SDDC Manager unable to perform any password operations on NSX-T Managers, with the error: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}
      As admin user: set auth-policy api lockout-period 0 && set auth-policy api lockout-reset-period 0