Subset of NSX Connected VMs Lost All Network Connectivity.

Products

• VMware NSX

Issue/Introduction

• Multiple production VMs unreachable or unable to traverse north of T0
• The affected VMs is successfully able to ping its gateway but cannot reach any address northbound of T0
• The following log messages are appearing in NSX-Manager <var/log/syslog> 
  

  <timestamps> UTC <NSX_MGR-FQDN>.au NSX 2387 - [nsx@6876 comp="nsx-controller" errorCode="CCP1310211" level="ERROR" subcomp="magpie"] Parallel invocation of features encountered error with concurrent listeners: Feature Manager didn't complete within timeout: Owl-worker-51#012 java.lang.Thread.State: WAITING#012 stack trace: #012 at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)#012 at [email protected]/java.util.concurrent.locks.LockSupport.park(Unknown Source)#012 at [email protected]/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)#012 at [email protected]/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)#012
  

• NSX Manager memory usage alarms may be present on all 3 NSX Managers
• NSX Manager uptime is greater than 26 days

Environment

VMware NSX 4.2.0.x, 4.2.1.0, 4.2.1.1, 4.2.1.2, 4.2.1.3 and 9.0.0 and below are impacted by this issue.

Cause

• This is a known issue documented in KB:396719

Resolution

• Reboot all 3 NSX Managers one at a time.
  • Wait for all services to show as "green" in NSX Manager UI before proceeding with next Manager reboot.
• For more information, please see main JDK KB NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks Due to ctl Field Release Count (RC) Overflow