Credential operations fail on NSX Components in SDDC Manager
search cancel

Credential operations fail on NSX Components in SDDC Manager

book

Article ID: 314657

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

This article helps troubleshoot failed credential operations with NSX-T Components. When attempting to rotate, update, or remediate a password for an NSX 4.x or NSX-T component in the SDDC Manager UI, you get error:

"Password management operation failed"

In the operationsmanager.log you see similar errors:
YYYY-MM-DDTHH:MM:SS.sssZ DEBUG [vcf_om,xxxxxxxxxxxxx,abcd] [c.v.v.p.helper.NsxtApiUtil,om-exec-5] Failed to get NSXT user details : {"module_name":"com
mon-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403} with status : 
YYYY-MM-DDTHH:MM:SS.sssZ ERROR [vcf_om,xxxxxxxxxxxxx,abcd] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-5] The credentials were incorrect or the account specified has been locked.
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: The credentials were incorrect or the account specified


In the /var/log/vmware/vcf/lcm/lcm-debug.log the following message is found:

Exception occurred during NSX API invocation java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized)
=> {messages = [],data = struct => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},errorType = UNAUTHORIZED}

Environment

  • VMware Cloud Foundation 9.x
  • VMware Cloud Foundation 5.x
  • VMware Cloud Foundation 4.x
  • VMware Cloud Foundation 3.x

Cause

This issue could be caused by the following:
  • NSX Manager passwords have expired.
  • NSX Manager passwords have been change manually outside of SDDC.

Because of this, it is required to manually set the password on NSX back to what is stored in SDDC. Once this has been completed, password operations for NSX in the SDDC manager will work again

Resolution

Pull the most recent passwords from the SDDC Manager

  • lookup_passwords
  • API Explorer

Steps for the NSX-T Managers
Only needs to be performed on one manager per cluster. 

  1. Log in to the NSX-T manager as root. (Either from a console window or SSH)
  2. Clear password history

    echo "" >/etc/security/opasswd

  3. Run the command

    /etc/init.d/nsx-mp-api-server stop

  4. Set the password(s) to match what is present in SDDC DB.

    passwd
    passwd admin
    passwd audit

  5. Run command:
    touch /var/vmware/nsx/reset_cluster_credentials

  6. Run the command:
    /etc/init.d/nsx-mp-api-server start
  7. Verify the accounts are not locked out with pam_tally2 or faillock

        For VCF versions 4.3.0.0 up till 5.0.0.1, use:

    pam_tally2 -u root -r
    pam_tally2 -u admin -r
    pam_tally2 -u audit -r

        For VCF versions starting from VCF 5.1.0.0, use:

        /usr/sbin/faillock --user root --reset
    /usr/sbin/faillock --user admin --reset
    /usr/sbin/faillock --user audit --reset

     8.  Retry the credential operation from the SDDC Manager UI. 

      

Steps for the NSX-T Edges
  1. Log in to the NSX-T edge as root. (Either from a console window or SSH)
  2. Clear password history:

    echo "" >/etc/security/opasswd

  3. Run the command:
    /etc/init.d/nsx-edge-api-server stop

  4. Set the password(s) to match what is present in SDDC DB.

    passwd
    passwd admin
    passwd audit

  5. Run the command:
    touch /var/vmware/nsx/reset_cluster_credentials

  6. Run the command:
    /etc/init.d/nsx-edge-api-server start

  7. Verify the accounts are not locked out with pam_tally2 or faillock
    For VCF versions 4.3.0.0 up till 5.0.0.1, use:

        pam_tally2 -u root -r
    pam_tally2 -u admin -r
    pam_tally2 -u audit -r

         For VCF versions starting from VCF 5.1.0.0, use:

    /usr/sbin/faillock --user root --reset
    /usr/sbin/faillock --user admin --reset
    /usr/sbin/faillock --user audit --reset

    8. Retry the credential operation from the SDDC Manager UI.

 

Steps to change password expiration on NSX-T edges and Managers:

  1. Connect to the NSX-T Manager or NSX-T Edge with the admin account.
    You can elevate to admin from a root connection with su admin or st en.

  2. Reset the expiration period.
    1. You can clear the expiration as the admin user:

      clear user admin password-expiration
      clear user root password-expiration
      clear user audit password-expiration

    2. Set the expiration period for between 1 and 9999 days.
nsxtmgr> set user admin password-expiration 9999
nsxtmgr> set user audit password-expiration 9999
nsxtmgr> set user root password-expiration 9999

 

Additional Information

  • Check with the following command to ensure the passwords got changed on the manager(s):
chage -l root
chage -l admin 
chage -l audit
  • Check to see if there's any locks:
curl http://localhost/locks | json_pp > releaseLock.json
curl -X PUT -H "Content-Type:application/json" http://localhost/locks -d @releaseLock.json
2021-04-09T13:41:44.830+0000 INFO  [vcf_lcm,0000000000000000,0000,precheckId=########-####-####-####-25e8fb993243,resourceType=NSX_T,resourceId=nsx.test.local] [c.v.e.s.l.p.i.nsxt.NsxtPrimitiveImpl,pool-3-thread-49] Completed precheck task NSX_T_PASSWORD_VALIDITY_CHECK  on resource id nsx.test.local with status RED
Powered by Wolken Software