Unable to validate NSX Manager(s) datasource, error seen on GUI is 'Invalid Credentials'

Products

• VCF Operations for Networks

Issue/Introduction

• You are not able to validate the NSX data source
• Credentials for NSX data sources have been validated and used for SSH connection but does not validate the NSX data source
• NSX Version is 
  • VMware NSX 4.2.0.x
  • VMware NSX 4.2.1.0, 4.2.1.1, 4.2.1.2, 4.2.1.3
  • VMware NSX 9.0.0.0
• NSX managers uptime for nodes is 30+ days
• The NSX data sources was working previously with no issue

 

Environment

• VCF Operations for Networks 6.X.X
• VCF Operations for Networks 9.X
• VMware NSX 4.X
• VMware NSX 9.X

Cause

The issue occurs due to a NSX JDK bug (JDK-8330017) where the Release Count (RC) field in ForkJoinPool's internal control structure overflows. The RC value keeps decreasing until it reaches -32768, then overflows to +32767 (ForkJoinPool.MAX_CAP), causing the thread pool to stop executing tasks.

This affects different NSX services:

• Controller service - impacts network provisioning, firewall rules, and vMotion operations
• Upgrade Coordinator service - affects upgrade operations and causes OOM errors
• Corfu service - impacts data storage and retrieval operations

The issue accumulates over time and becomes apparent during configuration changes (upgrades, VM migrations) or when memory limits are reached.

Resolution

For environments running affected versions implement a preventative monthly rolling reboot schedule:

1. Reboot the first NSX Manager.
2. SSH to a Manager as admin user and check cluster health: get cluster status
3. When all services report up on all 3 NSX Manager nodes, reboot the next Manager.
4. Repeat steps 2-3 for the third Manager.

Additional Information

NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks Due to ctl Field Release Count (RC) Overflow