NSX-T Edge Nodes Disconnected in Password Manager on SDDC
search cancel

NSX-T Edge Nodes Disconnected in Password Manager on SDDC

book

Article ID: 316043

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:
  • In VCF 4.5 you may run into an issue after attempting a password rotation or immediately after upgrading where all the NSX Edge nodes become disconnected
  • NSX Edge node passwords are not expired 
  • SSH may be disabled on the NSX Edge nodes 
 
image.png
 


Environment

Vmware Cloud Foundation 4.5

Cause

The NSX nodes are coming out of sync when the rotate happens. Sometimes the nodes drop sync almost right after a recent upgrade to VCF 4.5 


Resolution

  • Make sure SSH is enabled on the NSX Nodes 
  • Manually reset the NSX Edge passwords to what lookup_passwords has the password set as for each disconnected node
  1. Get Password from SDDC: 
    • SSH into SDDC with vcf user
    • Type su to get into root and login with root credentials 
    • Type: lookup_passwords and enter a username and password for a user with the ADMIN role
    • Enter in the component type: NSX 
    • This will list out all the NSX credentials - Paste into Notepad ++ 
  2. Resetting the NSX Edge Root Password: 
    • Connect to the console of the appliance.
    • Reboot the system.
    • When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. If you wait too long and the boot sequence does not pause, you must reboot the system again.
    • Press e to edit the menu.
    • Enter the user name root and the GRUB password for root (not the same as the appliance's user root).
    • Press e to edit the selected option.
    • Search for the line starting with linux and add systemd.wants=PasswordRecovery.service to the end of the line.
    • Press Ctrl-X to boot.
    • When the log messages stop, enter the new password for root.
    • Enter the password again.
    • The boot process continues.
    • After the reboot, you can verify the password change by logging in as root with the new password.
  3. Resetting the NSX Edge Aduit and Admin Passwords: 
    • Log in to the appliance as root.
    • For an NSX Intelligence appliance or a Cloud Service Manager, skip this step. For NSX Edge, run the command /etc/init.d/nsx-edge-api-server stop. Otherwise, run the command /etc/init.d/nsx-mp-api-server stop
    • (Optional) To reset the password for admin, run the command passwd admin.
    • (Optional) To reset the password for audit, run the command passwd audit.
    • (Optional) For NSX-T Data Center 3.1.1, to reset a guest user password, run the command passwd guestusername
    • Run the command touch /var/vmware/nsx/reset_cluster_credentials
    • For NSX Edge, run the command /etc/init.d/nsx-edge-api-server start. Otherwise, run the command /etc/init.d/nsx-mp-api-server start
  • Remediate in SDDC 
    1. From SDDC select Administration> Security> Password Management 
    2. Then click the tab 'NSX EDGE' 
    3. Click the vertical ellipsis (three dots) next to the node whose password you are trying to remediate and click Remediate 
    4. Enter and confirm the password that was manually reset. Confirm it matches the password from lookup_passwords on SDDC
    5. Click Remediate 
  • You should be able to rotate the NSX Edge passwords (if  needed) and they should no longer show as disconnected in SDDC 


Additional Information