NSX credentials are not being synchronized between NSX Managers after manual password reset.
search cancel

NSX credentials are not being synchronized between NSX Managers after manual password reset.

book

Article ID: 317192

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • NSX-T is running version 4.1.X
  • When resetting a local user password following the steps in the Admin guide - resetting passwords on an appliance, the passwords are not being synchronized among the NSX Managers in the cluster.
  • You made sure you issued the command "touch /var/vmware/nsx/reset_cluster_credentials" but despite that, password is not synchronized.
  • Rolling reboot of the NSX managers does not resolve the issue.
  • Even after replacing the NSX Managers that are not taking the new password with new NSX Managers, the issue remains.
  • In the NSX Manager logs, we will see similar entries as shown below:

    In /var/log/nvpapi/api_server*:
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found

    In /var/log/syslog*:
    NSX 1690 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="<USERNAME>" level="INFO"] Updated local etc file entries for users: 10000
    NSX 1690 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="<USERNAME>" level="INFO"] Updated local etc file entries for users: 10000
    NSX 84773 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="<USERNAME>" level="INFO"] Updated local etc file entries for users: 0, 10000, 10002
    NSX 84773 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="<USERNAME>" level="INFO"] Updated local etc file entries for users: 10000

Environment

VMware NSX 4.1.X

Cause

After running the steps in resetting passwords on an appliance, the file /var/vmware/nsx/reset_cluster_credentials remains on the host used for reset.
This file causes password synchronization issues for future password changes.

Resolution

This issue is resolved in VMware NSX 4.2, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.


Workaround
Manually delete this file /var/vmware/nsx/reset_cluster_credentials. In order to do this, please follow below steps:

  1. After following steps in resetting passwords on an appliance . Please wait for at least 1 minute for password sync to happen.
  2. Verify root, admin and audit passwords are the same on all three nodes. For this, you can SSH to all 3 NSX Managers nodes using the admin, root and audit credentials and validate login is successful.
  3. Then, SSH to the host used for password reset and log in as root.
  4. Delete the file /var/vmware/nsx/reset_cluster_credentials with the command
rm /var/vmware/nsx/reset_cluster_credentials
  1. Restart the nsx-mp-api-server/nsx-edge-api-server as follows:
systemctl restart nsx-mp-api-server
  1. After the service restarts passwords can be changed from admin with the command
set user admin password
set user audit password
set user root password
  1. These passwords will synchronize as expected.

 

If you know the password for root but have forgotten the password for your local users, and the above plan doesn't fix the issue, we can use steps below to reset the password:

  1. SSH to NSX Managers as root.
  2. Stop the nsx-mp-api-server service:
    /etc/init.d/nsx-mp-api-server stop
  3. (Optional) To reset the password for admin, run the command
    passwd admin
  4. (Optional) To reset the password for audit, run the command 
    passwd audit
  5. (Optional) To reset a guest user password, run the command 
    passwd <guestusername>
  6. Run the command 
    touch /var/vmware/nsx/reset_cluster_credentials
  7. Restart the nsx-mp-api-server service:
    /etc/init.d/nsx-mp-api-server start
  8. Now, re-attempt to log in to the NSX-T manager using admin or audit account from all nodes.

Note: If unable to login with new password, then run Resync command from the Active GM - using admin account:
> start search resync all

Powered by Wolken Software