Root user account is disconnected in SDDC Manager password management.

Products

• VMware Cloud Foundation
• VMware NSX
• VMware Workspace ONE Intelligence SaaS Standalone Subscription per Device

Issue/Introduction

• NSX manager and Workspace Once user accounts such as root, admin, and audit users, in SDDC manager password management are showing as disconnected.
• When trying to remediate the password for each user, some may fail to be remediated and the user account remained disconnected. 
• When trying to login locally to NSX UI (https:// nsx-manager/login.jsp?local=true) with the last known credential from SDDC manager, a generic error message is displayed indicating the password is incorrect or user has been locked-out. 
• When trying to login to Workspace One locally, it fails with error password expired.
• When trying to remediate the password for root account, failing with node is not ready state error.

 

Environment

VMware Cloud Foundation

VMware NSX

Workspace One

Cause

Possible reasons for user account disconnection:

• User account password expired
• User account locked out due to too many failed logins
• User account password was manually updated from NSX and not rotated through SDDC manager
• SDDC manager does not have the correct NSX/Workspace One user passwords

Resolution

 To resolve the issue in NSX/Workspace One manager appliance, check the following options:

• Can log in as admin from the UI and all user passwords are working

  • Check user password expiration status: nsxcli> get user <local-user-name> password-expiration
  • If the user password is expired, this can be reset or simply disabling password-expiration can be used therefore the password no longer expires.
    • To reset user password when current password is known: set user <local-user-name> password 
    • To disable password expiration for a user: clear user <local-user-name> password-expiration
  • Verified user password no longer expired: get user <local-user-name> password-expiration
  • Test if local user can login through NSX UI or a SSH session.
  • Use "Remediate" option in SDDC password management page for each of the NSX local user account that has the password updated or cleared of password-expiration.

• Cannot log in as a local user on NSX UI or SSH, however, root password is still working

  • To reset any local user's password when root user password is working, please follow the first half of this Techdoc. 
  • Test if local user can login through NSX UI or a SSH session.
  • Use "Remediate" option in SDDC password management page for each of the NSX local user account with reset password.

•  Root user password is lost or incorrect

  • Admin user can reset a user's password only when the current password is known, otherwise, root user access is required to reset any user's passwords.
  • To reset root user password, please follow the bottom half of the instruction in this Techdoc.
  • After the root user password recovery, follow the first half of the instruction of the same Techdoc to reset all other local user passwords if needed.
  • Use "Remediate" option in SDDC password management page for each of the NSX local user account with reset password.

•  Root and admin user passwords are already updated, however, remediate passwords in SDDC manager still fails and admin user cannot login through NSX UI or a SSH session

  • The affected users have their passwords updated using above methods.
  • Admin user still cannot login to NSX UI using the reset credential, the error message indicates incorrect password or user is locked-out.
  • Root user can login through NSX manager VM console but not through a SSH session (if permitRootLogin is enabled for SSH).
  • Option 1, disable user lock-out policy
    • Use a NSX manager VM console log in as root user
    • root#: su admin
    • nsxcli> set auth-policy api lockout-period 0
    • nsxcli> set auth-policy cli lockout-period 0 
    • Verify local users can log in from SSH or NSX UI again
    • Use "Remediate" option in SDDC password management page for each of the NSX local user account with reset password.
    • Re-enable lock-out policy to default value: 
      • nsxcli> set auth-policy api lockout-period 900
      • nsxcli> set auth-policy cli lockout-period 900
  • Option 2, rolling reboot of NSX managers
    • After all NSX manager nodes are rebooted one at a time, confirm local users can log in through NSX UI or a SSH session
    • Use "Remediate" option in SDDC password management page for each of the NSX local user account with reset password.

• Workspace One Root User account password is expired.

  • Login to Workspace One locally and set the new password.
  • Connect to SDDC Manager UI.
  • Navigate to Password Management.
  • Select the Root account and Remediate the password using the new password set on the Workspace One.
• Restart the Log Insight service

• • Restart the Log Insight service using the following command:
    service loginsight start
  • Once the service is verified as running, remediate the root password through the SDDC management UI.

Additional Information

Impact/Risks:
NONE - The steps provided are safe to run with no downtime on any of the components involved.