Credential operations fail on NSX-T Components in SDDC Manager
search cancel

Credential operations fail on NSX-T Components in SDDC Manager

book

Article ID: 314657

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

This article helps troubleshoot failed credential operations with NSX-T Components. When attempting to rotate, update, or remediate a password for an NSX 4.x or NSX-T component in the SDDC Manager UI, the following error is observed:

"Password management operation failed"

In the operationsmanager.log file, the following errors are observed:
YYYY-MM-DDTHH:MM:SS.sssZ DEBUG [vcf_om,xxxxxxxxxxxxx,abcd] [c.v.v.p.helper.NsxtApiUtil,om-exec-5] Failed to get NSXT user details : {"module_name":"com
mon-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403} with status : 
YYYY-MM-DDTHH:MM:SS.sssZ ERROR [vcf_om,xxxxxxxxxxxxx,abcd] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-5] The credentials were incorrect or the account specified has been locked.
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: The credentials were incorrect or the account specified


In the /var/log/vmware/vcf/lcm/lcm-debug.log file, the following entry is observed:

Exception occurred during NSX API invocation java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized)
=> {messages = [],data = struct => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},errorType = UNAUTHORIZED}

Environment

  • VMware Cloud Foundation 9.x
  • VMware Cloud Foundation 5.x
  • VMware Cloud Foundation 4.x
  • VMware Cloud Foundation 3.x

Cause

This issue can be caused by the following:
  • NSX-T passwords have expired.
  • NSX-T passwords have been changed manually outside of SDDC.

Because of this, it is necessary to manually reset the password in NSX to match the one stored in SDDC. Once this has been completed, password operations for NSX in the SDDC manager will work again.

Resolution

Pull the most recent passwords from the SDDC Manager

NOTE: The last known NSX-T passwords stored in SDDC Manager must be used to reset the manager and/or edge passwords.

Steps for the NSX-T Managers
Only needs to be performed on one manager per cluster. 

  1. Log in to the NSX-T manager as root. (Either from a console window or SSH)
  2. Clear password history:

    echo "" >/etc/security/opasswd

  3. Run the command:

    /etc/init.d/nsx-mp-api-server stop

  4. Set the password(s) to match what is present in the SDDC DB.

    passwd
    passwd admin
    passwd audit

  5. Run command:
    touch /var/vmware/nsx/reset_cluster_credentials

  6. Run the command:
    /etc/init.d/nsx-mp-api-server start
  7. Verify the accounts are not locked out with pam_tally2 or faillock

        For VCF versions 4.3.0.0 up till 5.0.0.1, use:

    pam_tally2 -u root -r
    pam_tally2 -u admin -r
    pam_tally2 -u audit -r

        For VCF versions starting from VCF 5.1.0.0, use:

        /usr/sbin/faillock --user root --reset
    /usr/sbin/faillock --user admin --reset
    /usr/sbin/faillock --user audit --reset

     8.  Retry the credential operation from the SDDC Manager UI. 

      

Steps for the NSX-T Edges
  1. Log in to the NSX-T edge as root. (Either from a console window or SSH)
  2. Clear password history:

    echo "" >/etc/security/opasswd

  3. Run the command:
    /etc/init.d/nsx-edge-api-server stop

  4. Set the password(s) to match what is present in SDDC DB.

    passwd
    passwd admin
    passwd audit

  5. Run the command:
    touch /var/vmware/nsx/reset_cluster_credentials

  6. Run the command:
    /etc/init.d/nsx-edge-api-server start

  7. Verify the accounts are not locked out with pam_tally2 or faillock
    For VCF versions 4.3.0.0 up till 5.0.0.1, use:

        pam_tally2 -u root -r
    pam_tally2 -u admin -r
    pam_tally2 -u audit -r

         For VCF versions starting from VCF 5.1.0.0, use:

    /usr/sbin/faillock --user root --reset
    /usr/sbin/faillock --user admin --reset
    /usr/sbin/faillock --user audit --reset

    8. Retry the credential operation from the SDDC Manager UI.

 

Steps to change password expiration on NSX-T edges and Managers:

  1. Connect to the NSX-T Manager or NSX-T Edge with the admin account.
    Elevate to admin from a root connection with su admin or st en.

  2. Reset the expiration period.
    1. The expiration can be cleared as the admin user:

      clear user admin password-expiration
      clear user root password-expiration
      clear user audit password-expiration

    2. Set the expiration period between 1 and 9999 days.
nsxtmgr> set user admin password-expiration 9999
nsxtmgr> set user audit password-expiration 9999
nsxtmgr> set user root password-expiration 9999

 

Additional Information

  • Check with the following command to ensure the passwords got changed on the manager(s):
chage -l root
chage -l admin 
chage -l audit
  • Check to see if there's any locks:
curl http://localhost/locks | json_pp > releaseLock.json
curl -X PUT -H "Content-Type:application/json" http://localhost/locks -d @releaseLock.json
2021-04-09T13:41:44.830+0000 INFO  [vcf_lcm,0000000000000000,0000,precheckId=########-####-####-####-25e8fb993243,resourceType=NSX_T,resourceId=nsx.test.local] [c.v.e.s.l.p.i.nsxt.NsxtPrimitiveImpl,pool-3-thread-49] Completed precheck task NSX_T_PASSWORD_VALIDITY_CHECK  on resource id nsx.test.local with status RED
Powered by Wolken Software