Edge and ESXi Host Transport Nodes May Be in a Degraded State After NSX Upgrade

Products

• VMware NSX

Issue/Introduction

After upgrading NSX or reconnecting Transport Nodes to NSX Manager, the following symptoms may be observed:

• Edge and Host Transport Nodes report a degraded state:
• BFD tunnels on Edge or Host Transport Nodes are down
• Network connectivity issues between workloads connected via NSX logical switches
• Transport Node health check failures

To determine if this issue is affecting your environment, check the CCP logs on the NSX Manager:

grep "ForkJoinPool.commonPool" /var/log/cloudnet/nsx-ccp.log

If no entries containing "ForkJoinPool.commonPool" appear for 10 minutes or longer, the CCP's ForkJoinPool is likely in a blocked state and the control plane cannot function normally.

The NSX Manager node might have run into the JDK issue at any time but during steady state, when there is no configuration change, no issue might surface. The event that triggers the issue to show up is typically an upgrade or other configuration change.

Environment

• VMware NSX 4.2.0.x
• VMware NSX 4.2.1.x
• VMware NSX 9.0.0

Cause

This issue is caused by a known JDK bug (JDK-8330017) where Java ForkJoinPool incorrectly determines the total number of threads as over the limit. This causes new thread requests to be blocked, which results in the NSX Controller transaction processing thread becoming unresponsive.

When the ForkJoinPool thread count reaches its maximum limit, the Control Cluster Platform (CCP) service cannot process new requests, affecting Transport Node connectivity and tunnel establishment.

Resolution

For complete resolution steps and preventative measures, see NSX is impacted by JDK-8330017: ForkJoinPool stops executing tasks due to ctl field Release Count (RC) overflow.

Once you've addressed the immediate issue by following the steps in the linked article, verify that Transport Node status returns to normal in the NSX UI after all NSX Manager services report as up.