Credential operations fail on NSX-T Components in SDDC Manager
search cancel

Credential operations fail on NSX-T Components in SDDC Manager

book

Article ID: 314657

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

This article helps troubleshoot failed credential operations with NSX-T Components. When attempting to rotate, update, or remediate a password for an NSX 4.x or NSX-T component in the SDDC Manager UI, the following error is observed:

"Password management operation failed"


In the /var/log/vmware/vcf/operationsmanager/operationsmanager.log file, the following errors are observed:

YYYY-MM-DDTHH:MM:SS.sssZ DEBUG [vcf_om,#############,abcd] [c.v.v.p.helper.NsxtApiUtil,om-exec-5] Failed to get NSXT user details : {"module_name":"com mon-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403} with status :
YYYY-MM-DDTHH:MM:SS.sssZ ERROR [vcf_om,#############,abcd] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-5] The credentials were incorrect or the account specified has been locked. com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: The credentials were incorrect or the account specified


In the /var/log/vmware/vcf/lcm/lcm-debug.log file, the following entry is observed:

Exception occurred during NSX API invocation java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized)
=> {messages = [],data = struct => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},errorType = UNAUTHORIZED}

Cause

This issue can be caused by the following:
  • NSX-T passwords have expired.
  • NSX-T passwords have been changed manually outside of SDDC.

Because of this, it is necessary to manually reset the password in NSX to match the one stored in SDDC. Once this has been completed, password operations for NSX in the SDDC manager will work again.

Resolution

Pull the most recent passwords from the SDDC Manager

NOTE: The last known NSX-T passwords stored in SDDC Manager must be used to reset the manager and/or edge passwords.

Steps for the NSX-T Managers
Only needs to be performed on one manager per cluster. 

  1. Log in to the NSX-T manager as root. (Either from a console window or SSH)
  2. Run the command:

    /etc/init.d/nsx-mp-api-server stop

  3. Clear password history:

    echo "" >/etc/security/opasswd

  4. Set the password(s) to match what is present in the SDDC DB.

    passwd
    passwd admin
    passwd audit

  5. Run command:

    touch /var/vmware/nsx/reset_cluster_credentials

  6. Run the command:

    /etc/init.d/nsx-mp-api-server start


  7. Verify the accounts are not locked out with pam_tally2 or faillock

        For VCF versions 4.3.0.0 up till 5.0.0.1, use:

    pam_tally2 -u root -r
    pam_tally2 -u admin -r
    pam_tally2 -u audit -r

        For VCF versions starting from VCF 5.1.0.0, use:

        /usr/sbin/faillock --user root --reset
    /usr/sbin/faillock --user admin --reset
    /usr/sbin/faillock --user audit --reset

     8.  Retry the credential operation from the SDDC Manager UI. 

      

Steps for the NSX-T Edges
  1. Log in to the NSX-T edge as root. (Either from a console window or SSH)
  2. Run the command:

    /etc/init.d/nsx-edge-api-server stop

  3. Clear password history:

    echo "" >/etc/security/opasswd

  4. Set the password(s) to match what is present in SDDC DB.

    passwd
    passwd admin
    passwd audit

  5. Run the command:

    touch /var/vmware/nsx/reset_cluster_credentials

  6. Run the command:

    /etc/init.d/nsx-edge-api-server start

  7. Verify the accounts are not locked out with pam_tally2 or faillock
    For VCF versions 4.3.0.0 up till 5.0.0.1, use:

        pam_tally2 -u root -r
    pam_tally2 -u admin -r
    pam_tally2 -u audit -r

         For VCF versions starting from VCF 5.1.0.0, use:

    /usr/sbin/faillock --user root --reset
    /usr/sbin/faillock --user admin --reset
    /usr/sbin/faillock --user audit --reset

    8. Retry the credential operation from the SDDC Manager UI.

 

Steps to change password expiration on NSX-T edges and Managers:

  1. Connect to the NSX-T Manager or NSX-T Edge with the admin account.
    Elevate to admin from a root connection with su admin or st en.

  2. Reset the expiration period.
    1. The expiration can be cleared as the admin user:

      clear user admin password-expiration
      clear user root password-expiration
      clear user audit password-expiration

    2. Set the expiration period between 1 and 9999 days.

      nsxtmgr> set user admin password-expiration 9999 nsxtmgr> set user audit password-expiration 9999 nsxtmgr> set user root password-expiration 9999

Additional Information

  • Check with the following command to ensure the passwords got changed on the manager(s):

    chage -l root
    chage -l admin 
    chage -l audit
  • Check to see if there's any locks:

    curl http://localhost/locks | json_pp > releaseLock.json curl -X PUT -H "Content-Type:application/json" http://localhost/locks -d @releaseLock.json
Powered by Wolken Software