NSX Transport Nodes show as disconnected or in an unknown state due to CCP deadlock

Products

• VMware NSX

Issue/Introduction

Transport Nodes show as disconnected in the UI or in an unknown state via the API. This issue occurs because Central Control Plane (CCP) and the affected Transport Nodes are unable to resync, which prevents the Transport Nodes from receiving configuration updates.

 

In /var/log/cloudnet/nsx-ccp.log, a repeated resync timeout for the affected Transport Nodes may be observed:

 

YYYY-MM-DDTHH:MM:SS.SSSZ   INFO CommonDelayedScheduler NestDbPigeonImpl 74430 - Calling StubEventListener to handle MerkleTree Resync error for receiver <UUID>

 

Additionally, in /var/log/cloudnet/nsx-ccp-wrapper.errput.*, a sign of ForkJoinPool thread starvation may be observed in the form of one or more long-running daemons parked on "NestDbPigeonImpl" running a "ReduceTask":

 

"ForkJoinPool-1-worker-6" #255 daemon prio=5 os_prio=0 cpu=11.76ms elapsed=707960.58s tid=0x000072436c6f65c0 nid=0x1d319 waiting on condition  [0x00007243660c9000]

java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park([email protected]/Native Method)

- parking to wait for  <0x0000724410613240> (a java.util.stream.ReduceOps$ReduceTask)

at java.util.concurrent.locks.LockSupport.park(java.base@17.0.18/Unknown Source)

at java.util.concurrent.ForkJoinTask.awaitDone(java.[email protected]/Unknown Source)

at java.util.concurrent.ForkJoinTask.invoke(java.base@17.0.18/Unknown Source)

at java.util.stream.ReduceOps$ReduceOp.evaluateParallel([email protected]/Unknown Source)

at java.util.stream.AbstractPipeline.evaluate([email protected]/Unknown Source)

at java.util.stream.ReferencePipeline.collect([email protected]/Unknown Source)

at com.vmware.nsx.pigeon.NestDbDataManagerImpl.getDataForReceiver(NestDbDataManagerImpl.java:274)

at com.vmware.nsx.pigeon.NestDbPigeonImpl.resyncWithMerkleTree(NestDbPigeonImpl.java:208)

Environment

• NSX 4.2.#
• NSX 9.0.#
• NSX 9.1.0
• NSX 9.1.1

Cause

A deadlock occurs within the Central Control Plane (CCP), leading to thread starvation in the ForkJoinPool. This prevents the CCP and TransportNodes from successfully resyncing, stalling the delivery of configurations to the TransportNodes

Resolution

Fixed in release 9.2.0 and higher.

Workaround:

• Perform controller service restart on the NSX manager node(s) which are impacted
  
  • In Admin of the NSX manager run the following:
    • restart service controller
    • get cluster status
    • get service controller
• Rebooting the NSX managers, if restarting the controller service doesn't work.