Unable to Rotate NSX Passwords from SDDC Manager Due to Expired Admin/Audit Credentials

Products

VMware NSX VMware SDDC Manager

Issue/Introduction

You may encounter an issue where attempting to rotate NSX Manager passwords from the SDDC Manager interface is unsuccessful. In this scenario, you will typically observe the following symptoms:

NSX Managers appear as disconnected from the SDDC Manager.
You are unable to change the passwords for the admin and audit accounts through the standard interface.
You still have access to the root password and can successfully log in to the NSX-T Manager using VCF SSO.

Environment

VMware Cloud Foundation (VCF) 9.x
VMware NSX

Cause

This issue occurs because the passwords for the NSX accounts (specifically the admin account) have expired. Communication between the SDDC Manager and NSX relies on the admin account for API interactions. Because the credentials have expired, API authentication fails, resulting in the disconnected state and preventing the SDDC Manager from automatically rotating the passwords.

Resolution

Documented process for resetting of passwords needs to be followed as per Resetting the Passwords of an Appliance

To resolve this issue, you must manually reset the passwords from the backend OS using the root account.

NOTE: You must know the current root password for the NSX Manager.

1.Log in to the NSX VIP using SSH as the root user. (Has to be VIP for it to propagate changes properly)

2.Stop the NSX Manager API service
Run the following command to stop the NSX Manager API service:

/etc/init.d/nsx-mp-api-server stop

2. the accounts have been locked out due to repeated failed API attempts from the SDDC Manager, clear the counters for both the admin and audit users:

1.faillock --user admin --reset
2. faillock --user audit --reset

3.Reset the admin password as you would on any standard Linux system as root:

root@nsx-mgr:/# passwd admin

or as admin run the below command to change the password: (For Guest Users and additionally created users you need to use passwd command as the below command will not work).

nsx-mgr> set user admin password

4.Create the reset_cluster_credentials file to trigger the necessary configuration update:

touch /var/vmware/nsx/reset_cluster_credentials

5.Restart the NSX Manager API service to apply the changes:

/etc/init.d/nsx-mp-api-server start

6. Attempt to log in to the NSX Manager UI again. The login should now be successful.

7. Remediate the associated user on SDDC
> Navigate back to Password Management in SDDC Manager.
> Locate the admin and audit accounts that are showing as "Disconnected".
>Click the three dots next to each and select Remediate.
>Enter the new passwords you just established via the OS shell.

After completing these steps, attempt to log in to the NSX Manager UI again. The login should now be successful.

Additional Information

If they don't know the root password, they can reset it via the GRUB:

Connect to the console of the appliance.
Reboot the system.
When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. If you wait too long and the boot sequence does not pause, you must reboot the system again.
Press e to edit the menu.
1. Choose the top Ubuntu line then enter the user name root and the GRUB password for root (not the same as the appliance's user root). The default password is NSX@VM!WaR10
Press e to edit the selected option.
Search for the line starting with linux and add systemd.wants=PasswordRecovery.service to the end of the line.
Press Ctrl-X to boot.
When the log messages stop, enter the new password for root.
Enter the password again.
1. The boot process continues.
After the reboot, you can verify the password change by logging in as root with the new password.

Additional KB reference: https://knowledge.broadcom.com/external/article/314637/root-user-account-is-disconnected-in-sdd.html