NSX Config multiple full sync issue due to IDS signature streaming failure
Article ID: 381159
Updated On: 11-04-2024
Products
Issue/Introduction
Setup observing multiple full sync due to IDS signature streaming failure.
You must see error like below on NSX logs(/var/log/proton/nsxapi.log)
2024-10-24T06:46:57.150Z ERROR intelligence-message-processor IntelligenceNsxCommunicationServiceImpl 5274 INTELLIGENCE [nsx@6876 comp="nsx-manager" errorCode="PM91905" level="ERROR" subcomp="manager"] Failed to send config message updates to NSX Intelligence.
com.vmware.nsx.pace.common.exceptions.PaceAgentException: null
at com.vmware.nsx.management.policy.paceagent.services.IntelligenceNsxCommunicationServiceImpl.send(IntelligenceNsxCommunicationServiceImpl.java:654) ~[?:?]
at com.vmware.nsx.management.policy.paceagent.services.IntelligenceNsxCommunicationServiceImpl.processMessages(IntelligenceNsxCommunicationServiceImpl.java:554) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
OR
2024-10-21T09:48:20.731Z ERROR intelligence-message-processor IntelligenceNsxCommunicationServiceImpl 152910 INTELLIGENCE [nsx@6876 comp="nsx-manager" errorCode="PM91905" level="ERROR" subcomp="manager"] Failed to send config message updates to NSX Intelligence.
2024-10-21T12:00:20.240Z ERROR IDS_AUTO_DOWNLOAD_TASK-0 IdsSignatureUtils 152910 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM523931" level="ERROR" subcomp="manager"] IDS - Got Error while downloading Signature Bundle from NSX Intel Cloud
at com.vmware.nsx.management.policy.ids.utils.IDSRestClient.executePost(IDSRestClient.java:118) ~[?:?]
at com.vmware.nsx.management.policy.ids.utils.PolicyIDSUtils.registerCloudCacheClient(PolicyIDSUtils.java:582) ~[?:?]
at com.vmware.nsx.management.policy.ids.utils.PolicyIDSUtils.downloadSignatures(PolicyIDSUtils.java:839) ~[?:?]
at com.vmware.nsx.management.policy.ids.utils.IdsSignatureUtils.downloadSignatureBundleFromCloud(IdsSignatureUtils.java:243) ~[?:?]
at com.vmware.nsx.management.policy.ids.utils.IdsSignatureUtils.isRepoUpToDate(IdsSignatureUtils.java:166) ~[?:?]
at com.vmware.nsx.management.policy.ids.utils.IdsSignatureUtils.startDownload(IdsSignatureUtils.java:116) ~[?:?]
at com.vmware.nsx.management.policy.ids.utils.PolicyIDSAutoDownloadTask.run(PolicyIDSAutoDownloadTask.java:29) ~[?:?]
2024-10-21T12:00:20.244Z INFO IDS_AUTO_DOWNLOAD_TASK-0 IdsSignatureUtils 152910 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] IDS updating status with Download status as ERROR and signature status as UNAVAILABLE
2024-10-21T12:00:20.248Z WARN IDS_AUTO_DOWNLOAD_TASK-0 MultiVersionObject 152910 SnapshotProxy[308] encountered trimmed addresses [] during sync to 343031 on attempt 1 of 2
2024-10-21T12:00:20.254Z INFO IDS_AUTO_DOWNLOAD_TASK-0 ImmutableCorfuTable 152910 ImmutableCorfuTable: creating PersistentCorfuTable with the following indexes: HashSet(targets.path_prefix, abstract_policy_resource.owner_id)
2024-10-21T12:00:20.258Z INFO IDS_AUTO_DOWNLOAD_TASK-0 PolicyServiceImpl 152910 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Entity /infra/settings/firewall/security/intrusion-services/signatures/status does not exist, creating
2024-10-21T12:00:20.260Z WARN IDS_AUTO_DOWNLOAD_TASK-0 MultiVersionObject 152910 SnapshotProxy[3f47] encountered trimmed addresses [] during sync to 343031 on attempt 1 of 2
2024-10-21T12:00:20.265Z INFO IDS_AUTO_DOWNLOAD_TASK-0 ImmutableCorfuTable 152910 ImmutableCorfuTable: creating PersistentCorfuTable with the following indexes: HashSet(abstract_policy_resource.owner_id)
2024-10-21T12:00:20.280Z WARN IDS_AUTO_DOWNLOAD_TASK-0 MultiVersionObject 152910 SnapshotProxy[3bd3] encountered trimmed addresses [] during sync to 343032 on attempt 1 of 2
2024-10-21T12:00:20.284Z INFO IDS_AUTO_DOWNLOAD_TASK-0 ImmutableCorfuTable 152910 ImmutableCorfuTable: creating PersistentCorfuTable with the following indexes: HashSet(applied_to_entity, applied_to_standalone_hosts, applied_to_target, abstract_policy_resource.owner_id)
2024-10-21T12:00:20.291Z INFO IDS_AUTO_DOWNLOAD_TASK-0 IdsSignatureUtils 152910 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] IDPS is disabled so clear the alarm if its present
2024-10-21T12:00:20.291Z ERROR IDS_AUTO_DOWNLOAD_TASK-0 IdsSignatureUtils 152910 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM523933" level="ERROR" subcomp="manager"] IDS - Got Exception while checking whether the Repo is upto date or not.
Environment
Security Intelligence : 4.2.0
NSX : 4.2.1
Cause
Alarm gets raised for Config agent unhealthy on NSX and new config updates will not be reflected in the Discover and Take Action Page. The IDS signatures downloaded from NTICS now include multiple hierarchies of dependent signatures along with information about the signature. This leads to an increase size of the IDS signature and impacts the config not being available on NAPP UI.
Resolution
1. Scale down nsx-config to 0 replica :
napp-k -n nsxi-platform scale statefulset nsx-config --replicas=0
We can verifiy replica set value using below command.
napp-k get statefulsets nsx-config -o yaml | grep replica
2. Drain nsx2pace-config kafka topic
Get cluster-api pod name: napp-k get pod | grep cluster-api
Login to cluster-api container : napp-k exec -i cluster-api-XXXXXX -- /bin/bash
Run command to reset offsets: '/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server kafka:9092 --command-config /root/adminclient.props --reset-offsets --group intelligence-nsx-config-update --topic nsx2pace-config --to-latest --execute'
3. Remove IdsSignatureVersion table from TOI(Table Of Interest) list
Call NSX config-streaming-agent get consumer API to get current TOI
curl --location --request GET 'https://<nsx-manager-ip>/policy/api/v1/config-streaming-agent/consumers/NAPP'
Above GET request will give NAPP Consumer details in response, copy response body and send it as request body in Consumer Update request as per below instruction
Send Update config-streaming-agent Consumer request
curl --location --request PUT 'https://<nsx-manager-ip>/policy/api/v1/config-streaming-agent/consumers/NAPP' \
--header 'Content-Type: application/json' \
--data-raw '<consumer-json-here>'
Use Consumer JSON received from 3.a in request body, search IdsSignatureVersion table and remove it from the tables list.
{
"name": "IdsSignatureVersion",
"versions": [
"CLASSIC"
]
},
4. Scale up nsx-config to 1 replica >>
napp-k -n nsxi-platform scale statefulsets nsx-config --replicas=1
Feedback
