NSX VM workloads impacted by "JDK ForkJoinPool issue" post ESXi patching/upgrade.
search cancel

NSX VM workloads impacted by "JDK ForkJoinPool issue" post ESXi patching/upgrade.

book

Article ID: 407647

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • VMs lost network connectivity on newly patched/upgraded ESXi host transport node.
    • No errors reported on NSX UI, ESXi host shows connected to the manager and controller.
      • On ESXi host nsxcli, following commands "get controllers" and "get managers" will result as connected.
    • All the services are up and running on corresponding ESXi host Transport Node.
    • vMotion of VMs connected to NSX segments on the patched ESXi host will not report any issues. vMotion task get completed successfully.
    • But no segments or logical switches are realized on ESXi Host post patching/Upgrade. This results in the network outage for the VMs connected to NSX segments.
      • On ESXi host nsxcli, following commands will not show any information (empty output) related to segments and logical switches.
      • ESXi host# nsxcli
        ESXi host> get segments
                              Segments Summary
        ------------------------------------------------------------
                            Overlay Kernel Entry
        ============================================================
          VNI                    DVS name                 VIF num
                             Overlay LCP Entry
        ============================================================
          VNI                 Segments UUID                 Name
                             VLAN Backed Entry
        ============================================================
                     Segments UUID               VLAN ID
        ESXi host> get logical-switches
                          Logical Switches Summary
        ------------------------------------------------------------
                            Overlay Kernel Entry
        ============================================================
          VNI                    DVS name                 VIF num
                             Overlay LCP Entry
        ============================================================
          VNI              Logical Switch UUID              Name
                             VLAN Backed Entry
        ============================================================
                  Logical Switch UUID             VLAN ID
  • Log references:
    • On NSX manager in the log file location /var/log/cloudnet/nsx-ccp-events.log states that ESXi Host transport node completing the handshake with CCP but not finishing full sync.
      • cat /var/log/cloudnet/nsx-ccp-events.log | grep -i "accepts incoming"
        2025-07-20T00:11:13.552Z EVENT nsx-rpc:CCP-<CCP UUID>:user-executor-2 VersionMastershipServiceImpl 73820 - [nsx@6876 comp="nsx-controller" level="EVENT" subcomp="handshake"] Accepts incoming connection from TN <TN UUID>
    • On NSX manager, the concerned ESXi Host TNs does not complete full syncs due to the known "Java 11 commonPool" issue.
      • <NSX manager>#cat /var/log/cloudnet/nsx-ccp.log | grep -i "ForkJoinPool.commonPool" | grep -i "2025-07-20T0" | grep -i "<TN UUID>"
          --------<No result for the above validation, No entry's reported confirms JDK issue>--------
        <NSX manager>#cat /var/log/cloudnet/nsx-ccp.log | grep -i "ForkJoinPool.commonPool" | grep -i "2025-07-20T1" | grep -i "<TN UUID>"
          --------<No result for the above validation, No entry's reported confirms JDK issue>--------

Environment

VMware NSX 4.2.1.1

Cause

  • During the ESXi Host patching process, the ESXi Host TNs get disconnected from CCP, and upon the attempt to reconnect with CCP, it cannot sync due to the existing JDK issue  (JDK-8330017).
  • This, in turn causes the ESXi Host Transport Node to get into the loop of Un-Synced Realization Tracking Framework in the log location /var/log/proton/nsxapi.log 
NSX Manager # cat var/log/proton/nsxapi.log | grep -i "2025-07-20T00:1" | grep -i "Returning current realization status"
2025-07-20T00:12:31.329Z  INFO com.vmware.nsx.management.policy.policyframework.realization.StatusTracker RealizationStateServiceImpl 76694 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Returning current realization status 'Status = 'UNKNOWN', Message = 'Unable to collect status for '2' transport nodes.', TNs = '[TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>, Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'UNKNOWN', Message = 'Transport node is not connected to CCP.[Controller <Controller UUID>: This CCP is not the master of this TN.][Controller <Controller UUID>: This CCP is not the master of this TN.]', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = <TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'UNKNOWN', Message = 'Transport node is not connected to CCP.[Controller <Controller UUID>: This CCP is not the master of this TN.][Controller <Controller UUID>: This CCP is not the master of this TN.]', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'., TN = '<TN UUID>', Status = 'SUCCESS', Message = '', Errors = '[]'.]', Pending Changes = '[]'.' for entity 'FirewallSection/ffffffff-<Framework entity ID>'.
2025-07-20T00:15:16.346Z  INFO com.vmware.nsx.management.policy.policyframework.realization.StatusTracker RealizationStateServiceImpl 76694 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Returning current realization status 'Status = 'IN_PROGRESS', Message = 'There are no transport nodes in the span.', TNs = '[]', Pending Changes = '[]'.' for entity 'LogicalSwitch/<LSwitch ID>'.

Resolution

For resolution and workaround, refer to the parent article that consolidates guidance regarding this issue: NSX is impacted by JDK-8330017: ForkJoinPool stops executing tasks due to ctl field Release Count (RC) overflow.