NSX management plane instability causing disconnected credentials or licensing failures in SDDC Manager

Products

• VMware NSX

Issue/Introduction

• SDDC Manager Password Manager displays NSX admin credentials as Disconnected.
• VMware Cloud Foundation (VCF) license upgrade tasks remain in a Stuck or Failed state.
• The SDDC Manager UI shows a persistent deployment lock.
• NSX Manager VIP or UI is unreachable or intermittently responsive.
• NSX overlay tunnels are down on transport nodes.
• NSX Management plane instability may result in service unresponsiveness, specifically impacting the ability to update vIDM SSL Thumbprints or causing disconnected credentials within SDDC Manager.
• Error when attempting to update the vIDM SSL Thumbprint (referencing KB 386982).
• SDDC Manager Password Manager displays NSX admin credentials as Disconnected.
• VMware Cloud Foundation (VCF) license upgrade tasks remain in a Stuck or Failed state.
• NSX Manager VIP or UI is unreachable or intermittently responsive.

Environment

• VMware NSX 4.2.0.x
• VMware NSX 4.2.1.3 and earlier
• VMware NSX 9.0.0

Cause

A JDK bug (JDK-8330017) causing thread pool overflows can lead to a degraded state in the NSX management cluster. This prevents stable communication between SDDC Manager and the NSX VIP, causing synchronization failures and stale database locks.

For more information, see NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks Due to ctl Field Release Count (RC) Overflow

Resolution

Fixed in release VMware NSX 4.2.1.4, 4.2.2, 9.0.1.0, and higher. See Download Broadcom products and software for steps to download these releases.

Broadcom recommends a rolling reboot of NSX Managers prior to upgrading to a fixed release version to avoid potential problems associated with this issue. For environments running affected versions, implement a preventative monthly rolling reboot schedule:

1. Stabilize NSX Management Cluster:
   • Reboot the first NSX Manager.
   • SSH to the rebooted NSX Manager as admin user and check cluster health: get cluster status
   • When all services report UP on all three NSX Manager nodes, reboot the next Manager.
   • Repeat steps for the remaining NSX Manager nodes.
2. Clear SDDC Manager Database Tasks:
   • Log in to SDDC Manager via SSH and switch to root.
   • Update stuck password tasks to USER_CANCELLED in the operationsmanager database.
3. Remove Stale Deployment Locks:
   • Execute the following command in the platform database:
     
     psql -h localhost -U postgres -d platform -c "delete from lock where resource_type='deployment';"
      
   • Run /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh to synchronize the environment state.

Additional Information

• Note: Rebooting the NSX Manager that owns the VIP causes a brief disruption in connectivity to the Cluster VIP. Reboot the node that owns the VIP last.
• If tunnels on transport nodes remain down following a rolling reboot, restart services on the transport nodes to restore connectivity:
  1. SSH to the affected ESXi host.
  2. Restart NSX services:
     
     /etc/init.d/nsx-opsagent restart  /etc/init.d/nsx-proxy restart
     
     Restarting the service or rebooting the NSX Manager node resolves immediate symptoms. However, without upgrading to a fixed version, the problem will recur over time.