VM has no network connectivity due to a blocked port after power on or vmotion
search cancel

VM has no network connectivity due to a blocked port after power on or vmotion

book

Article ID: 391622

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • NSX UI may display Cluster Degraded alarm
  • Newly created VMs, or recently migrated/vmotion'd VMs do not connect to the network
  • On the vSphere Client, Networking -> vDS Name -> Ports, the impacted VM port is in a "Blocked" state.
  • New configuration changes, such as segment updates, policy updates, etc, are delayed or blocked.
  • This condition may occur in two ways.

    • Scenario #1: The Controller service transaction processing thread is blocked but automatically self-recovers after 2 hours (7200 seconds).
      Log lines similar to the below are encountered on the NSX Manager in /var/log/cloudnet/nsx-ccp.log 
      ERROR FalconThread-0 AbstractDependencyBasedDataDiscoverer 74130 - [nsx@6876 comp="nsx-controller" errorCode="CCP1310211" level="ERROR" subcomp="magpie"] Parallel invocation of features encountered error with concurrent listeners: {}
      java.util.concurrent.TimeoutException: Shutdown timer hit after 7200 seconds
       Log lines similar to the below are encountered on the NSX Manager in /var/log/cloudnet/nsx-ccp-events.log 
      EVENT WrapperSimpleAppMain Main 2200380 - [nsx@6876 comp="nsx-controller" level="EVENT" subcomp="main"] CCP process started
    • Scenario #2: The Controller service service transaction processing thread is blocked indefinitely and does not self-recover.
      No log lines containing "ForkJoinPool.commonPool" are seen for 30 mins or longer on the NSX Manager in /var/log/cloudnet/nsx-ccp.log 
      INFO ForkJoinPool.commonPool-worker-63 ShardingManagerImpl 87947 - [nsx@4413 comp="nsx-controller" level="INFO" subcomp="magpie"] Notify listeners for sharding update with revision ########

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX 4.2.0.x
VMware NSX 4.2.1.x

Cause

Due to an issue in JDK (JDK-8330017), Java ForkJoinPool may incorrectly determine the total number of ForkJoinPool threads as over the limit and new thread requests may be blocked, which results in the NSX Controller transaction processing thread being blocked.
There are two possible scenarios:

  • Scenario #1: The Controller service is impacted however it auto restarts after 2 hours to self recover. VMs requesting a new network connection after a vmotion or power on are impacted during this 2 hour window.
  • Scenario #2: ForkJoinPool.commonPool may become blocked and the Controller service cannot recover without a manual restart. VMs requesting a new network connection after a vmotion or power on are impacted until the issue is manually resolved.

Note, this issue is expected to repeat based on the uptime of the Controller service. Medium form factor Managers can experience the issue after 6 weeks and Large/Extra Large form factor Managers after more than 3 months.

Resolution

Powered by Wolken Software