Credential operations fail on NSX Components in SDDC Manager

Products

• VMware Cloud Foundation

Issue/Introduction

This article helps troubleshoot failed credential operations with NSX-T Components. When attempting to rotate, update, or remediate a password for a NSX-T component in the SDDC Manager UI, you get error:

"Password management operation failed"

In the operationsmanager.log you see similar errors:

2023-03-28T20:29:42.487+0000 DEBUG [vcf_om,e76c9c17e51fce97,50f3] [c.v.v.p.helper.NsxtApiUtil,om-exec-5] Failed to get NSXT user details : {"module_name":"com
mon-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403} with status :
2023-03-28T20:29:42.509+0000 ERROR [vcf_om,e76c9c17e51fce97,50f3] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-5] The credentials were incorrect or the account specified has been locked.
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: The credentials were incorrect or the account specified

Environment

• VMware Cloud Foundation 5.0.x
• VMware Cloud Foundation 4.0.x
• VMware Cloud Foundation 3.0.x

Cause

This issue could be caused by the following:

• NSX-T passwords have expired.
• NSX-T passwords have been change manually outside of SDDC.

Resolution

Pull the most recent passwords from the SDDC Manager

• lookup_passwords
• API Explorer

Steps for the NSX-T Managers
Only needs to be performed on one manager per cluster. 

1. Log into the NSX-T manager as root. (Either from a console window or SSH)
2. Clear password history
   
   echo "" >/etc/security/opasswd
   
   
3. Run the command
   
   /etc/init.d/nsx-mp-api-server stop
   
   
4. Set the password(s) to match what is present in SDDC DB.
   
   passwd
passwd admin
passwd audit
   
   
5. Run command: touch /var/vmware/nsx/reset_cluster_credentials
6. Run the command: /etc/init.d/nsx-mp-api-server start
7. Verify the accounts are not locked out with pam_tally2
   
   pam_tally2 -u root -r
pam_tally2 -u admin -r
pam_tally2 -u audit -r
   
   
8. Retry the credential operation from the SDDC Manager UI. 

Steps for the NSX-T Edges

1. Log into the NSX-T edge as root.(Either from a console window or SSH)
2. Clear password history:
   
   echo "" >/etc/security/opasswd
   
   
3. Run the command: /etc/init.d/nsx-edge-api-server stop
4. Set the password(s) to match what is present in SDDC DB.
   
   passwd
passwd admin
passwd audit
   
   
5. Run the command: touch /var/vmware/nsx/reset_cluster_credentials
6. Run the command: /etc/init.d/nsx-edge-api-server start
7. Verify the accounts are not locked out with pam_tally2
   
   pam_tally2 -u root -r
pam_tally2 -u admin -r
   
   
8. Retry the credential operation from the SDDC Manager UI. 

Steps to change password expiration on NSX-T edges and Managers:

1. Connect to the NSX-T Manager or NSX-T Edge with the admin account.
   You can elevate to admin from a root connection with su admin OR st en.
2. Reset the expiration period.
   1. You can clear the expiration as the admin user:
      
      clear user admin password-expiration
clear user root password-expiration
clear user audit password-expiration
      
      
   2. Set the expiration period for between 1 and 9999 days.

nsxtmgr> set user admin password-expiration 9999
nsxtmgr> set user audit password-expiration 9999
nsxtmgr> set user root password-expiration 9999

Additional Information

• Check to see if there's any locks:

curl http://localhost/locks | json_pp > releaseLock.json
curl -X PUT -H "Content-Type:application/json" http://localhost/locks -d @releaseLock.json

• SDDC Manager unable to perform any password operations on NSX-T Managers, with the error: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403} (88561)
• If the password has expired and you're not able to reset it from the CLI/Console you'll have to reset it from the GRUB. See Resetting the Passwords of an Appliance.
• SDDC Manager password operations are not allowed because of a failed password task (90716)
• SDDC manager falsely shows the passwords of NSX-T components as expired during precheck/upgrade.
  
  Log file /var/log/vmware/vcf/lcm/lcm-debug.log will show following entries :

2021-04-09T13:41:44.830+0000 INFO  [vcf_lcm,0000000000000000,0000,precheckId=e8ef23bd-5e65-44c6-80b0-25e8fb993243,resourceType=NSX_T,resourceId=nsx.test.local] [c.v.e.s.l.p.i.nsxt.NsxtPrimitiveImpl,pool-3-thread-49] Completed precheck task NSX_T_PASSWORD_VALIDITY_CHECK  on resource id nsx.test.local with status RED

• • Executing the command "get user admin password-expiration" on NSX-T Manager may show as  "Password expiration not configured for this user".
    • Cause: Password expiry is disabled and/or is set to 'never expire' (admin) or '-1' (root) for the NSX-T component, the precheck will falsely show the password as expired or the schedule rotation may fail.
    • Workaround: Set the password expiry to 9999 or a lesser number by using the command "set user admin password-expiration 9999" (as admin) OR 'chage -M 9999 root' (as root) before performing precheck/upgrade or remediation/rotation.
      • If remediation fails, set the lockout to 0 by following KB: SDDC Manager unable to perform any password operations on NSX-T Managers, with the error: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403} 
        As admin user: set auth-policy api lockout-period 0 && set auth-policy api lockout-reset-period 0