NSX Edge Nodes Disconnected in Password Manager on SDDC

Products

VMware Cloud Foundation VMware NSX

Issue/Introduction

In VCF 4.5 and later, you may run into an issue after attempting a password rotation or immediately after upgrading.
The status of NSX Edge and NSX Manager may be shown as disconnected under Password Management of SDDC Manager.
NSX Edge node passwords are not expired

In the /var/log/vmware/vcf/lcm/lcm-debug.log the following message is found:

Exception occurred during NSX API invocation java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized)
=> {messages = [],data = struct => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},errorType = UNAUTHORIZED}

The NSX edge cluster name was changed directly from the NSX.

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Cause

There are two potential Causes:

The NSX nodes are coming out of sync when the rotation happens. Sometimes the nodes drop sync almost right after a recent upgrade of VCF.
The NSX cluster name was changed outside the SDDC.

Resolution

Make sure SSH is enabled on the NSX Nodes.
Manually reset the NSX Edge passwords to what lookup_passwords has the password set as for each disconnected node

Get Password from SDDC:
- SSH into SDDC with vcf user
- Type su to get into root and login with root credentials
- Type: lookup_passwords and enter a username and password for a user with the ADMIN role
- Enter the component type: NSXManager / NSXEdge
- This will list out all the NSX credentials. Copy them to a notepad.
Resetting the NSX Edge Root Password:

If the root password is known:

On the NSX UI, go to System > Fabric > Nodes, select the Edge, Actions > Change Credentials

If the root password is unknown:
- Connect to the console of the appliance from the vSphere client or Host client
- Reboot the system.
- When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. If you wait too long and the boot sequence does not pause, you must reboot the system again.
- Press e to edit the menu.
- Enter the user name root and the GRUB password for root (not the same as the appliance's user root).
- Press e to edit the selected option.
- Search for the line starting with linux and add systemd.wants=PasswordRecovery.service to the end of the line.
- Press Ctrl-X to boot.
- When the log messages stop, enter the new password for root.
- Enter the password again.
- The boot process continues.
- After the reboot, you can verify the password change by logging in as root with the new password.
Resetting the Audit and Admin Passwords for NSX Edge and NSX Manager

If the Edge admin or audit password is known:

On the NSX UI, go to System > Fabric > Nodes, select the Edge, Actions > Change Credentials, select cli for admin or Audit for audit

If the Manager admin password is known:

On the NSX UI, go to System > User Management > Local Users
Select the admin user and Change Password

If the Manager audit password is known or unknown:

On the NSX UI, go to System > User Management > Local Users
For audit user, select Reset Password

If the current passwords are unknown:
- Log in to the appliance as root.
- For an NSX Intelligence appliance or a Cloud Service Manager, skip this step. For NSX Edge, run the command /etc/init.d/nsx-edge-api-server stop
- To reset the password for admin, run the command passwd admin
- To reset the password for audit, run the command passwd audit
- (Optional) For NSX-T Data Center 3.1.1, to reset a guest user password, run the command passwd guestusername
- Run the below commands once the password is reset
  - touch /var/vmware/nsx/reset_cluster_credentials
  - /etc/init.d/nsx-edge-api-server start
    - set auth-policy cli lockout-period 0
    - set auth-policy cli max-auth-failures 0
- For NSXT manager , run the command /etc/init.d/nsx-mp-api-server stop
- To reset the password for admin, run the command passwd admin.
- Run the below commands once the password is reset
  - touch /var/vmware/nsx/reset_cluster_credentials
  - /etc/init.d/nsx-mp-api-server start
  - set auth-policy api lockout-period 0
  - set auth-policy api lockout-reset-period 0
  - set auth-policy api max-auth-failures 0

Remediate the passwords in SDDC Manager:

1. From SDDC navigate to Administration> Security> Password Management
2. Then click the tab 'NSX EDGE' / NSX MANAGER
3. Click the vertical ellipsis (three dots) next to the node whose password you are trying to remediate, and click Remediate.
4. Enter and confirm the password that was manually reset. Confirm it matches the password from lookup_passwords on SDDC
5. Click Remediate
You should be able to rotate the NSX Edge passwords (if needed) and they should no longer show as disconnected in SDDC
If the Edge cluster name was changed outside the SDDC, use the KB NSX Edge node password update in SSDC manager fails with an error "Unable to find edge cluster with name "edge test client"

Additional Information

Check with the following command to ensure the passwords have been changed on the manager(s):
```
chage -l root
chage -l admin 
chage -l audit
```

If it doesn't show the passwords were recently modified, refer to KB: NSX credentials are not being synchronized between NSX Managers after manual password reset.