NSX Manager nodes Failed to boot with error "failed to start default target"
search cancel

NSX Manager nodes Failed to boot with error "failed to start default target"

book

Article ID: 435436

calendar_today

Updated On:

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

  • Some or all VMware NSX Manager nodes fail to boot and display the following error message on the console screen:

    "failed to start default target: transaction for nsx-custom.target/start is destructive (emergency.target has 'start' job queued, but 'stop' is included in transaction)"
  • The output of the command "get cluster status" shows the NSX Manager cluster status as "Unknown"
  • In some scenarios. the NSX UI is inaccessible with error: Some appliance components are not functioning properly. 
    Component health: SEARCH:UNKNOWN, MANAGER:UNKNOWN, NODE_MGMT:UP, UI:UP.
    Error code: 101
  • The /config and the /non-config are in normal utilization 1-2%
  • All services are running normally, and the Corfu Compactor Tool runs every 15 minutes without gaps.

  • Running the File System Check (fsck) tool does not resolve the issue.

Environment

VMware NSX

Cause

An environmental disruption, such as a storage/vSAN outage or a network outage, can cause corruption within the NSX Manager appliance file system or state.

A standard three-node NSX Manager cluster continues to be fully functional if a single node is lost. However, if two nodes are lost, the cluster loses its quorum (majority) and will no longer be operational. With only one remaining node unable to establish a majority, access to the UI is lost.

Resolution

The official and supported procedure to recover an operational NSX node cluster is to deploy a new appliance, restore the environment from a valid backup, and then add two additional nodes to rebuild the cluster.

If a restore from backup is not immediately possible, use the following conditional workflows to attempt recovery depending on the state of the remaining manager nodes:

Scenario 1a: At least one NSX manager nodes are still running, NSX UI is available when connecting to the node's IP/FQDN

  1. Access the NSX UI and navigate to System > Appliances.

  2. Locate the problematic NSX Manager node and initiate the deletion process.

  3. Once the problematic node is successfully removed, deploy a new node directly from the same interface page.

Scenario 1b: At least one NSX manager node are still running, however, NSX UI is unavailable on any manager nodes

  1. If only one NSX Manager node is showing as unavailable, follow the standard NSX product documentation to detach the nodes showing as unavailable.

     

  2. If two NSX Managers or the entire cluster show as unavailable, follow the standard NSX product documentation to deactivate the NSX Manager cluster.

  3. Check if the admin shell outputs the status of the surviving node as up or down (instead of unavailable) by running:

    • nsx-manager> get cluster status
  4. Shut down the problematic node VMs.

  5. Reboot the surviving NSX Manager node.

  6. Run the get cluster status command on the surviving Manager node and wait for all services to transition to UP.

  7. Once the NSX UI is confirmed to be operational, delete the broken manager node VMs from vCenter Server.

  8. Access the NSX UI and re-deploy NSX managers 2 and 3.

  9. If the NSX UI encounters errors when deploying additional nodes, deploy the new manager nodes using OVA file.

  10. Join the newly deployed NSX manager to the surviving node/cluster
  11. Verify that the NSX Manager cluster status displays as stable under System > Appliances.

Scenario 2a: All NSX manager nodes are not booting to login shell but, backup is available

Scenario 2b: All NSX manager nodes are not booting to login shell and no valid backup was made and available for the NSX managers 

  • Broadcom Support does not have a mechanism to restore an NSX environment if no valid backup is available.

  • If a restore from backup is impossible, open a support case referencing this KB article to have the situation assessed for potential alternative recovery options. For more information, see Creating and managing Broadcom support cases.

Additional Information

Recovering an NSX Manager cluster that has lost 2 nodes

NSX Manager fails to boot with error "start default target: operation refused, unit may not be isolated"