NSX Manager fails to boot with error "start default target: operation refused, unit may not be isolated"

Products

• VMware NSX

Issue/Introduction

• After attempting to install unauthorized or "outside the box" applications and third-party packages on an NSX Manager node, the node goes down or becomes unresponsive.
  
  
• Rebooting the node fails and the manager node may show the following error message on the VM console:
  

Environment

VMware NSX
VMware NSX-T Datacenter

Cause

The NSX Manager is a closed appliance. Installing third-party applications or packages are not supported and causes the NSX manager appliance corruption. 

Resolution

There are several ways to restore NSX manager cluster depending on if there is still at least one manager node remains working:

Scenario 1a: At least one NSX manager nodes are still running, NSX UI is available when connecting to the node's IP/FQDN

• Access the NSX UI and go to System, Appliances. 
• Locate the problem NSX Manager node, then initiate the deletion.
• Once the problem node is removed successfully, a new node can be deploy from the same page.

Scenario 1b: At least one NSX manager node are still running, however, NSX UI is unavailable on any manager nodes

• If there is only one NSX Manager showing as unavailable
  • Follow this guide and detach nodes showing unavailable
• If there are two NSX Managers or the entire cluster showed as unavailable
  • Follow this guide to deactivate NSX manager cluster
• Check if the following command from the admin shell outputs the status of the surviving node as up or down (instead of unavailable).
  • • nsx-manager> get cluster status
• Shut down the problem nodes.
• Reboot the surviving NSX Manager node.
• Run the above "get cluster status" command on the surviving Manager node and wait for all the services to show as "UP". 
• Once the NSX UI is confirmed working, delete the broken manager node VMs from vCenter. 
• Access the NSX UI and re-deploy NSX managers 2 and 3.
• If NSX UI has errors when deploying the additional nodes, deploy the new manager nodes using OVA file.
  • Join the newly deployed NSX manager to the surviving node/cluster
• Verify the NSX manager cluster is now showing "stable" under System > Appliance.

Scenario 2a: All NSX manager nodes are not booting to login shell but, backup is available

• Deploy a brand new NSX manager using OVA file and restore backup from the SFTP server. 

Scenario 2b: All NSX manager nodes are not booting to login shell and no valid backup was made and available for the NSX managers 

• Please note that Broadcom Support do not have a way to restore your NSX environment if no valid backup is available.
• Please raise a Support Request with Broadcom Support to assist attempts of recovering the manager nodes.