NSX Config multiple full sync issue due to IDS signature streaming failure
search cancel

NSX Config multiple full sync issue due to IDS signature streaming failure

book

Article ID: 381159

calendar_today

Updated On:

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Setup observing multiple full sync due to IDS signature streaming failure.

You must see error like below on NSX logs(/var/log/proton/nsxapi.log) 

2024-10-24T06:46:57.150Z ERROR intelligence-message-processor IntelligenceNsxCommunicationServiceImpl 5274 INTELLIGENCE [nsx@6876 comp="nsx-manager" errorCode="PM91905" level="ERROR" subcomp="manager"] Failed to send config message updates to  NSX Intelligence.
com.vmware.nsx.pace.common.exceptions.PaceAgentException: null
        at com.vmware.nsx.management.policy.paceagent.services.IntelligenceNsxCommunicationServiceImpl.send(IntelligenceNsxCommunicationServiceImpl.java:654) ~[?:?]
        at com.vmware.nsx.management.policy.paceagent.services.IntelligenceNsxCommunicationServiceImpl.processMessages(IntelligenceNsxCommunicationServiceImpl.java:554) ~[?:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]

OR

 

2024-10-21T09:48:20.731Z ERROR intelligence-message-processor IntelligenceNsxCommunicationServiceImpl 152910 INTELLIGENCE [nsx@6876 comp="nsx-manager" errorCode="PM91905" level="ERROR" subcomp="manager"] Failed to send config message updates to NSX Intelligence.
2024-10-21T12:00:20.240Z ERROR IDS_AUTO_DOWNLOAD_TASK-0 IdsSignatureUtils 152910 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM523931" level="ERROR" subcomp="manager"] IDS - Got Error while downloading Signature Bundle from NSX Intel Cloud
        at com.vmware.nsx.management.policy.ids.utils.IDSRestClient.executePost(IDSRestClient.java:118) ~[?:?]
        at com.vmware.nsx.management.policy.ids.utils.PolicyIDSUtils.registerCloudCacheClient(PolicyIDSUtils.java:582) ~[?:?]
        at com.vmware.nsx.management.policy.ids.utils.PolicyIDSUtils.downloadSignatures(PolicyIDSUtils.java:839) ~[?:?]
        at com.vmware.nsx.management.policy.ids.utils.IdsSignatureUtils.downloadSignatureBundleFromCloud(IdsSignatureUtils.java:243) ~[?:?]
        at com.vmware.nsx.management.policy.ids.utils.IdsSignatureUtils.isRepoUpToDate(IdsSignatureUtils.java:166) ~[?:?]
        at com.vmware.nsx.management.policy.ids.utils.IdsSignatureUtils.startDownload(IdsSignatureUtils.java:116) ~[?:?]
        at com.vmware.nsx.management.policy.ids.utils.PolicyIDSAutoDownloadTask.run(PolicyIDSAutoDownloadTask.java:29) ~[?:?]
2024-10-21T12:00:20.244Z  INFO IDS_AUTO_DOWNLOAD_TASK-0 IdsSignatureUtils 152910 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] IDS updating status with Download status as ERROR and signature status as UNAVAILABLE
2024-10-21T12:00:20.248Z  WARN IDS_AUTO_DOWNLOAD_TASK-0 MultiVersionObject 152910 SnapshotProxy[308] encountered trimmed addresses [] during sync to 343031 on attempt 1 of 2
2024-10-21T12:00:20.254Z  INFO IDS_AUTO_DOWNLOAD_TASK-0 ImmutableCorfuTable 152910 ImmutableCorfuTable: creating PersistentCorfuTable with the following indexes: HashSet(targets.path_prefix, abstract_policy_resource.owner_id)
2024-10-21T12:00:20.258Z  INFO IDS_AUTO_DOWNLOAD_TASK-0 PolicyServiceImpl 152910 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Entity /infra/settings/firewall/security/intrusion-services/signatures/status does not exist, creating
2024-10-21T12:00:20.260Z  WARN IDS_AUTO_DOWNLOAD_TASK-0 MultiVersionObject 152910 SnapshotProxy[3f47] encountered trimmed addresses [] during sync to 343031 on attempt 1 of 2
2024-10-21T12:00:20.265Z  INFO IDS_AUTO_DOWNLOAD_TASK-0 ImmutableCorfuTable 152910 ImmutableCorfuTable: creating PersistentCorfuTable with the following indexes: HashSet(abstract_policy_resource.owner_id)
2024-10-21T12:00:20.280Z  WARN IDS_AUTO_DOWNLOAD_TASK-0 MultiVersionObject 152910 SnapshotProxy[3bd3] encountered trimmed addresses [] during sync to 343032 on attempt 1 of 2
2024-10-21T12:00:20.284Z  INFO IDS_AUTO_DOWNLOAD_TASK-0 ImmutableCorfuTable 152910 ImmutableCorfuTable: creating PersistentCorfuTable with the following indexes: HashSet(applied_to_entity, applied_to_standalone_hosts, applied_to_target, abstract_policy_resource.owner_id)
2024-10-21T12:00:20.291Z  INFO IDS_AUTO_DOWNLOAD_TASK-0 IdsSignatureUtils 152910 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] IDPS is disabled so clear the alarm if its present
2024-10-21T12:00:20.291Z ERROR IDS_AUTO_DOWNLOAD_TASK-0 IdsSignatureUtils 152910 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM523933" level="ERROR" subcomp="manager"] IDS - Got Exception while checking whether the Repo is upto date or not.

Environment

Security Intelligence : 4.2.0 

NSX : 4.2.1 

 

Cause

Alarm gets raised for Config agent unhealthy on NSX and new config updates will not be reflected in the Discover and Take Action Page. The IDS signatures downloaded from NTICS now include multiple hierarchies of dependent signatures along with information about the signature. This leads to an increase size of the IDS signature and impacts the config not being available on NAPP UI. 

Resolution

1. Scale down nsx-config to 0 replica :

napp-k -n nsxi-platform scale statefulset nsx-config --replicas=0 

We can verifiy replica set value using below command. 

napp-k get statefulsets nsx-config -o yaml | grep replica 

 

2. Drain nsx2pace-config  kafka topic

Get cluster-api pod name:  napp-k get pod | grep cluster-api  

Login to cluster-api container :  napp-k exec -i cluster-api-XXXXXX  -- /bin/bash 

Run command to reset offsets:   '/opt/kafka/bin/kafka-consumer-groups.sh  --bootstrap-server kafka:9092 --command-config /root/adminclient.props --reset-offsets --group intelligence-nsx-config-update --topic nsx2pace-config --to-latest --execute'

 

3. Remove IdsSignatureVersion table from TOI(Table Of Interest) list

Call NSX config-streaming-agent get consumer API to get current TOI

curl --location --request GET 'https://<nsx-manager-ip>/policy/api/v1/config-streaming-agent/consumers/NAPP'

Above GET request will give NAPP Consumer details in response, copy response body and send it as request body in Consumer Update request as per below instruction

Send Update config-streaming-agent Consumer request

curl --location --request PUT 'https://<nsx-manager-ip>/policy/api/v1/config-streaming-agent/consumers/NAPP' \
--header 'Content-Type: application/json' \
--data-raw '<consumer-json-here>'
Use Consumer JSON received from 3.a in request body, search IdsSignatureVersion table and remove it from the tables list.
{
    "name": "IdsSignatureVersion",
    "versions": [
        "CLASSIC"
    ]
},


4. Scale up nsx-config to 1 replica >>
      napp-k -n nsxi-platform scale statefulsets nsx-config --replicas=1