search cancel

Data Aggregator sync failure due to item with NULL domainID

book

Article ID: 255164

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

CAPM is failing to display data.  System Status show data aggregator with 'synchronization failure'.  'Test' of connection is fine.  Attempted 'full sync', it also fails.  Error in 'karaf.log' shows:

WARN  | 1958859436-34764 | 2022-11-28T09:36:31,731 | PhaseInterceptorChain | ache.cxf.common.logging.LogUtils  465 | org.apache.cxf.cxf-core |       | Application {http://netqos.com/ProductSync2WS}IProductSync2WSService#{http://netqos.com/ProductSync2WS}PullRequest has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: null
...
Caused by: java.lang.NullPointerException
 at com.ca.im.connector.productsync2.PullCustomItemTypeHandler.addAddresses(PullCustomItemTypeHandler.java:372) ~[bundleFile:?]
 at com.ca.im.connector.productsync2.PullCustomItemTypeHandler.processBatchedItem(PullCustomItemTypeHandler.java:340) ~[bundleFile:?]
 at com.ca.im.connector.productsync2.PullCustomItemTypeHandler.processBatchedItem(PullCustomItemTypeHandler.java:43) ~[bundleFile:?]
 at com.ca.im.connector.productsync2.PullStageHandlerBase.getNextBatch(PullStageHandlerBase.java:172) ~[bundleFile:?]
 at com.ca.im.connector.productsync2.PullSyncRequest.handleStage(PullSyncRequest.java:269) ~[bundleFile:?]
 at com.ca.im.connector.productsync2.PullSyncRequest.execute(PullSyncRequest.java:229) ~[bundleFile:?]
 at com.ca.im.connector.productsync2.ProductSync2WS.pullRequest(ProductSync2WS.java:778) ~[bundleFile:?]
 at jdk.internal.reflect.GeneratedMethodAccessor738.invoke(Unknown Source) ~[?:?]
 at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
 at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[bundleFile:?]
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[bundleFile:?]
 at com.sun.proxy.$Proxy359.pullRequest(Unknown Source) ~[?:?]
 at jdk.internal.reflect.GeneratedMethodAccessor738.invoke(Unknown Source) ~[?:?]
 at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
 at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
 at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) ~[bundleFile:3.5.0]
 at org.apache.cxf.jaxws.JAXWSMethodInvoker.performInvocation(JAXWSMethodInvoker.java:66) ~[bundleFile:3.5.0]
 at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) ~[bundleFile:3.5.0]
 ... 62 more

Attempted restarts of 'dadaemon' and 'activemq' on DA server, along with 'device manager' on Portal have not helped.

DX NetOps Performance Management Portal web UI shows the Data Aggregator (DA) Data Source (DS) failing to synchronize. No reports work due to the DA sync failure. This happened after starting an integrated VNA environment running an NSX-T plugin.

Environment

All supported DX NetOps Performance Management releases

Cause

A virtual interface item has an invalid parent device item id, resulting in lookup of DomainID to fail and throw a NullPointerException causing the DA sync to fail.

Resolution

To prevent DA sync failure when faced with an item with NULL for domainID code change will be introduced to the Data Aggregator. The current ETA is to have the new code included with the 22.2.6 release, though that is subject to change. The code change work is being done via defect ID DE552095. That will be used to reference this in the Fixed Issues list when published.

It will add code to the DA sync allowing it to report an error about these conditions while still successfully syncing. At some point in the future work will be done in the VNA code to address it's feeding of items that trigger this condition to the DA. This is the latest information as of the latest update to this article on January 23rd, 2023.

To resolve the issue in the existing environment take the following steps.

  1. Run the following vsql to identify the offending items with no domainID set.
    • This assumes dauser is the schema name. Change that as needed.
    • select item_id, device_item_id from dauser.poll_item where item_id in (select item_id from dauser.v_item_facet where facet_qname='{http://im.ca.com/inventory}VirtualInterface') and not exists (select null from dauser.v_item where poll_item.device_item_id=v_item.item_id) and exists (select null from dauser.v_item where poll_item.item_id=v_item.item_id) order by 2,1;
    • In the sample output below we see 6 items.
      • Those with no device_item_id associated wouldn't be synced to Portal, thus wouldn't trigger this failure.
      • Those with a device_item_id associated are the offending items.
  2. Delete the offending item using the following CURL statement.
    • Replace <user> with any Administrator Role assigned user.
    • Replace <DA_HOST> with the DA host name.
    • Replace <ID> with the ID identified in step 1.
    • Enter the password when prompted. 
    • curl -X DELETE -u <user> http://<DA_HOST>:8581/rest/syncable/<ID>
  3. Rerun the vsql command from step 1 to confirm the offending items are deleted.
  4. DA sync should now succeed.

Sample commands and output to ID offending items. Note here only item_id 18645461 is a problem. There may be more than one.

dradmin=> select item_id, device_item_id from dauser.poll_item where item_id in (select item_id from dauser.v_item_facet where facet_qname='{http://im.ca.com/inventory}VirtualInterface') and not exists (select null from dauser.v_item where poll_item.device_item_id=v_item.item_id) and exists (select null from dauser.v_item where poll_item.item_id=v_item.item_id) order by 2,1;
 item_id  | device_item_id 
----------+----------------
 18643259 |               
 18644735 |               
 18644736 |               
 18645459 |               
 18645460 |               
 18645461 |       16938094
(6 rows)

Sample CURL to remove it, run on the DA CLI.

(DA_HOST)# curl -X DELETE -u admin http://DA_HOST:8581/rest/syncable/18645461
Enter host password for user 'admin':
SUCCESS
(DA_HOST)#

The vsql command after the CURL DELETE is issued.

dradmin=> select item_id, device_item_id from dauser.poll_item where item_id in (select item_id from dauser.v_item_facet where facet_qname='{http://im.ca.com/inventory}VirtualInterface') and not exists (select null from dauser.v_item where poll_item.device_item_id=v_item.item_id) and exists (select null from dauser.v_item where poll_item.item_id=v_item.item_id) order by 2,1;
 item_id  | device_item_id 
----------+----------------
 18643259 |               
 18644735 |               
 18644736 |               
 18645459 |               
 18645460 |               
(5 rows)

Powered by Wolken Software