Intelligence recommendations fail to publish due to full sync not completing before next full sync begins

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Symptoms:
NSX Intelligence recommendations fail to publish.
When a full sync between NSX Intelligence & NSX Intelligence-agent does not complete within 5 minutes, another full sync request gets re-triggered from NSX-Intelligence and sometimes results in continuous full sync storms culminating in a config sync failure.

Use the following steps to check the nsx-config pod logs to figure out if it is due to the config sync issue.
 NOTE: All commands below are executed as root ont he CLI of the NSX Manager

Step 1: Find the name of nsx-config pod using below command 
napp-k get pods | grep nsx-config
 

Example output:

nsxi-platform metrics-nsx-config-cc8744c4d-2bvx5 1/1 Running 0 7d1h
nsxi-platform metrics-nsx-config-create-kafka-topic-frchf 0/1 Completed 0 7d4h
nsxi-platform nsx-config-85c776d6bb-sl66s 1/1 Running 0 7d1h
nsxi-platform nsx-config-create-kafka-topic-v4lbq 0/1 Completed 0 7d4h
 
 
Step 2: Copy the name of the nsx-config pod. Here the name is nsx-config-85c776d6bb-sl66s
 
Step 3: Execute below command replacing the pod name(i.e. nsx-config-85c776d6bb-sl66s) from step 2
 
napp-k logs <nsx-config-xxxxxx-xxx>  | grep 'fullsync checker thread didn't wakeup by full sync complete message'
 
 
If the output looks like below, this means there is a config sync issue
xxxx-xx-xxT08:13:31,789 INFO [Thread-8] c.v.n.p.n.s.FullSyncHandleServiceImpl$FullSyncChecker: INTELLIGENCE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] fullsync checker thread didn't wakeup by full sync complete message, will request for another full sync.

Environment

Impacted environments are the below:
NAPP 3.x version lower than 3.2.1
NAPP 4.x version lower than 4.1.1

Resolution

The below is require for 3.x and 4.x versions less than 3.2.1 and 4.1.1


1. Execute the below command and edit the property in configmap
napp-k edit configmap nsx-config

2. Enter in insert mode and change the fullSyncTimeoutMills value.

fullSyncTimeoutMills: 300000 ← Old value
fullSyncTimeoutMills: 1200000 ← New value

Workaround:
Delete the nsx-config pod with the below command replacing the pod name with the name from Step 2.

napp-k delete pod nsx-config-xxxxxx-xxx -n nsxi-platform

The nsx-config pod will be terminated and will be re-spun up automatically.

You can run napp-k get pod |grep nsx-config to verify the pod comes up and is in a running state

Additional Information

Impact/Risks:
This causes recommendations to fail to publish.