NCP stuck for a long time in the initialization of cache after an NCP restart in an NSX-T+PAS environment
search cancel

NCP stuck for a long time in the initialization of cache after an NCP restart in an NSX-T+PAS environment

book

Article ID: 321179

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:

  • NSX Container Plugin (NCP) stuck in long time in initialization of cache after an NCP restart. 
  • This initialization time depends on the actual app-instances scale on PAS environment.



Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.x

Cause

This issue occurs due to a large amount of upstream and downstream resources needed to sync up when initializing. Any new app push or recreation of app-instances fails as NCP is busy in initializing.

Resolution

This is a known issue affecting VMware NSX Container Plugin (NCP) 2.4.1.

This is fix in VMware NSX Container Plugin (NCP) 2.5.1 and onwards.

Workaround:
To work around this issue:

  1. Confirm from the logs that the NCP initialization completes and is ready to take a first app-instance request by running this command:

    grep 'ActualLRPController worker 0 started syncing' in /var/vcap/sys/log/ncp/ncp.stdout.log
     
  2. Another example is by running this command:

    tail -F ncp.stdout.log | grep 'ActualLRPController worker 0 started syncing'.

    You should see a similar response similar to:

    2019-09-08T08:02:05.966Z f4e7cec2-5306-4260-9955-de3678da73e0 NSX 10571 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="INFO"] nsx_ujo.common.controller ActualLRPController worker 0 started syncing (u'477b6bfc-689a-42a0-89a8-a9a6e199383c', u'3') 
     
  3. Once the "Initialized NCP cache" completes, re-stage the failed apps.