NSX credentials are not being synchronized between NSX Managers after manual password reset.
search cancel

NSX credentials are not being synchronized between NSX Managers after manual password reset.

book

Article ID: 317192

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • NSX-T is running version 4.1.X
  • When resetting a local user password following the steps in this doc Admin guide - Resetting the Passwords of an Appliance, the passwords are not being synchronized among the NSX Managers in the cluster.
  • You made sure you issued the command "touch /var/vmware/nsx/reset_cluster_credentials" but despite that, password is not synchronized.
  • Rolling reboot of the NSX managers does not resolve the issue.
  • Even after replacing the NSX Managers that are not taking the new password with new NSX Managers, the issue remains.
  • In the NSX Manager logs, we will see similar entries as shown below:

    In /var/log/nvpapi/api_server*:
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found
    napi.root.node.users.__self__ WARNING Any of shadow or passwd entry not found

    In /var/log/syslog*:
    NSX 1690 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="<USERNAME>" level="INFO"] Updated local etc file entries for users: 10000
    NSX 1690 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="<USERNAME>" level="INFO"] Updated local etc file entries for users: 10000
    NSX 84773 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="<USERNAME>" level="INFO"] Updated local etc file entries for users: 0, 10000, 10002
    NSX 84773 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="<USERNAME>" level="INFO"] Updated local etc file entries for users: 10000

Environment

VMware NSX 4.1.X

Cause

After running the steps in Resetting the Passwords of an Appliance, the file /var/vmware/nsx/reset_cluster_credentials remains on the host used for reset.
This file causes password synchronization issues for future password changes.

Resolution

This issue is resolved in VMware NSX 4.2, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.


Workaround
Manually delete this file /var/vmware/nsx/reset_cluster_credentials. In order to do this, please follow below steps:

  1. After following steps in Resetting the Passwords of an Appliance. Please wait for at least 1 minute for password sync to happen.
  2. Verify root, admin and audit passwords are the same on all three nodes. For this, you can SSH to all 3 NSX Managers nodes using the admin, root and audit credentials and validate login is successful.
  3. Then, SSH to the host used for password reset and log in as root.
  4. Delete the file /var/vmware/nsx/reset_cluster_credentials with the command
rm /var/vmware/nsx/reset_cluster_credentials
  1. Restart the nsx-mp-api-server/nsx-edge-api-server as follows:
systemctl restart nsx-mp-api-server
  1. After the service restarts passwords can be changed from admin with the command
set user admin password
set user audit password
set user root password
  1. These passwords will synchronize as expected.



Powered by Wolken Software