Deployment of Additional NSX Manager Stuck at ‘Deploying’ Stage in NSX Manager UI

Products

• VMware NSX

Issue/Introduction

When adding an additional NSX appliance from the NSX Manager UI using the Add NSX Appliance option, the deployment gets stuck at the ‘Deploying’ stage and does not progress further as shown in the following screenshot:

• Cluster status for MANAGER role is STABLE.
• The following logs in /var/log/proton/nsxapi.log, obtained using grep -A3 "LeaderBalancingHelper", indicate that an NSX Manager had no leadership for the MANAGER role despite not being in Maintenance Mode.

YYYY-MM-DDTHH:MM:45.412Z  INFO GmleClientBlockingOpsThread-2 LeaderBalancingHelper 53118 - [nsx@6876 comp="nsx-manager" level="INFO" s2comp="leader-balancing-helper" subcomp="manager"] ------------------- LEADERSHIP STATUS OF MEMBER XXX-XX-XX-XX-XXXXX OF GROUP XXX-XX-XX-XX-XXXXX--------
Category: SMALL :: Total weight: 0, Max capacity: 0, Current occupied weight: 0, holds leadership for services: {}
Category: MEDIUM :: Total weight: 0, Max capacity: 0, Current occupied weight: 0, holds leadership for services: {}
Category: LARGE :: Total weight: 0, Max capacity: 0, Current occupied weight: 0, holds leadership for services: {}

 

 

Environment

VMware NSX

Cause

Under normal conditions, when the ‘MANAGER’ status is STABLE (as shown in the cluster status output) and no nodes are in maintenance mode, each node should hold leadership for some services.

If a node is found without any leadership roles, it has likely encountered a race condition, causing the deployment to remain stuck at the ‘Deploying’ stage.

Resolution

This issue is resolved in VMware NSX 9.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

• Access the root CLI of the affected NSX Manager (the one without any leadership of service for group MANAGER).
• Stop the proton service: service proton stop
  Because the NSX Manager is not participating in the MANAGER cluster group, there's no impact to stop the proton service.
• Wait 1 minute.
• Start the proton service: service proton start