DX NetOps 21.2.x Maximum Number of Child Elements Exceeded Sync Failure

Products

• CA Network Flow Analysis (NetQos / NFA)
• CA Performance Management - Usage and Administration
• CA Spectrum

Issue/Introduction

After upgrading the DX NetOps 21.2.x you may notice some of your data sources fail to sync with the NetOps Portal. This can occur between DX NetOps Portal and Network Flow Analysis, Spectrum, or Data Aggegator. The error would appear like this in the DMService.log:

ERROR | pool-2-thread-2          | 2021-10-14 17:29:02,489 | com.ca.im.portal.dm.productsync.DataSourcePoller                 
      | 
Data source Network Flow [email protected] encountered an error while processing a sync request.  The problem is in the data source, not CAPC.  Check the logs for the data source to determine the cause of the problem.  The following stack trace shows the context of the sync request:
javax.xml.ws.soap.SOAPFaultException: Unmarshalling Error: Maximum Number of Child Elements limit (50000) Exceeded 
 at org.apache.cxf.jaxws.JaxWsClientProxy.mapException(JaxWsClientProxy.java:195)
 at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
 at com.sun.proxy.$Proxy114.pullRequest(Unknown Source)
 at com.ca.im.portal.api.services.datasource.ProductSyncWSProxyImpl.pullRequest(ProductSyncWSProxyImpl.java:74)
 at com.ca.im.portal.dm.productsync.DataSourcePull.pullSync(DataSourcePull.java:139)
 at com.ca.im.portal.dm.productsync.DataSourcePullPhase.executeInternal(DataSourcePullPhase.java:37)
 at com.ca.im.portal.dm.productsync.DataSourceSyncPhase.execute(DataSourceSyncPhase.java:76)
 at com.ca.im.portal.dm.productsync.PullSyncPhaseTask.pullDataSource(PullSyncPhaseTask.java:103)
 at com.ca.im.portal.dm.productsync.PullSyncPhaseTask.execute(PullSyncPhaseTask.java:70)
 at com.ca.im.portal.dm.productsync.SyncPhaseExecutor$MakeDSSyncPhaseCallable$1.call(SyncPhaseExecutor.java:94)
 at com.ca.im.portal.dm.productsync.SyncPhaseExecutor$MakeDSSyncPhaseCallable$1.call(SyncPhaseExecutor.java:87)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: javax.xml.bind.UnmarshalException
 - with linked exception:

From the NFA perspective, you will see no errors in the ProductSyncWsLog when you run into this issue. The only indication is from the NetOps Portal Data Sources page where it will show a sync failure, as well as the DMService log error above. 

Environment

Release : 21.2.x

Component : Network Flow Analysis, Spectrum, Data Aggregator, NetOps Portal

Cause

The out of the box values are not set high enough for some larger customer environments.

Resolution

From a command line accessing the DX NetOps Portal please edit and add these additional java parameters to replace the default value of 50000:

1. Edit PC/conf/wrapper-user.conf and add the following line:
wrapper.java.additional.33=-Dorg.apache.cxf.stax.maxChildElements=100000
 
2. Edit DM/conf/wrapper-user.conf and add the following line:
wrapper.java.additional.33=-Dorg.apache.cxf.stax.maxChildElements=500000

Restart the devicemanager and console services:

service caperfcenter_devicemanager stop
service caperfcenter_console stop
service caperfcenter_devicemanager start
service caperfcenter_console start