NSX-T Edge Nodes Disconnected in Password Manager on SDDC

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

In VCF 4.5 and later, you may run into an issue after attempting a password rotation or immediately after upgrading
The status of NSX Edge and NSX Manager may be shown disconnected under Password Management of SDDC Manager
NSX Edge node passwords are not expired
In the /var/log/vmware/vcf/lcm/lcm-debug.log the following message is found:

Exception occurred during NSX API invocation java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized)
=> {messages = [],data = struct => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},errorType = UNAUTHORIZED}

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Cause

The NSX nodes are coming out of sync when the rotate happens. Sometimes the nodes drop sync almost right after a recent upgrade of VCF.

Resolution

Make sure SSH is enabled on the NSX Nodes
Manually reset the NSX Edge passwords to what lookup_passwords has the password set as for each disconnected node

Get Password from SDDC:
- SSH into SDDC with vcf user
- Type su to get into root and login with root credentials
- Type: lookup_passwords and enter a username and password for a user with the ADMIN role
- Enter the component type: NSXManager / NSXEdge
- This will list out all the NSX credentials. Copy them to a notepad.
Resetting the NSX Edge Root Password:
- Connect to the console of the appliance from the vSphere client or Host client
- Reboot the system.
- When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. If you wait too long and the boot sequence does not pause, you must reboot the system again.
- Press e to edit the menu.
- Enter the user name root and the GRUB password for root (not the same as the appliance's user root).
- Press e to edit the selected option.
- Search for the line starting with linux and add systemd.wants=PasswordRecovery.service to the end of the line.
- Press Ctrl-X to boot.
- When the log messages stop, enter the new password for root.
- Enter the password again.
- The boot process continues.
- After the reboot, you can verify the password change by logging in as root with the new password.
Resetting the Audit and Admin Passwords for NSX Edge and NSX Manager
- Log in to the appliance as root.
- For an NSX Intelligence appliance or a Cloud Service Manager, skip this step. For NSX Edge, run the command /etc/init.d/nsx-edge-api-server stop
- To reset the password for admin, run the command passwd admin
- To reset the password for audit, run the command passwd audit
- (Optional) For NSX-T Data Center 3.1.1, to reset a guest user password, run the command passwd guestusername
- Run the below commands once the password is reset
  - touch /var/vmware/nsx/reset_cluster_credentials
  - /etc/init.d/nsx-edge-api-server start
    - set auth-policy cli lockout-period 0
    - set auth-policy cli max-auth-failures 0
- For NSXT manager , run the command /etc/init.d/nsx-mp-api-server stop
- To reset the password for admin, run the command passwd admin.
- Run the below commands once the password is reset
  - touch /var/vmware/nsx/reset_cluster_credentials
  - /etc/init.d/nsx-mp-api-server start
  - set auth-policy api lockout-period 0
  - set auth-policy api lockout-reset-period 0
  - set auth-policy api max-auth-failures 0

Remediate the passwords in SDDC Manager:
1. From SDDC navigate to Administration> Security> Password Management
2. Then click the tab 'NSX EDGE' / NSX MANAGER
3. Click the vertical ellipsis (three dots) next to the node whose password you are trying to remediate and click Remediate
4. Enter and confirm the password that was manually reset. Confirm it matches the password from lookup_passwords on SDDC
5. Click Remediate
You should be able to rotate the NSX Edge passwords (if needed) and they should no longer show as disconnected in SDDC

Additional Information

Check with the following command to ensure the passwords got changed on the manager(s):

chage -l root
chage -l admin 
chage -l audit

- If it doesn't show the passwords were recently modified refer to KB: NSX credentials are not being synchronized between NSX Managers after manual password reset.