NFA to Portal Circular Reference Synchronization failure

Products

CA Performance Management Network Flow Analysis Network Observability

Issue/Introduction

While looking at the NetOps Portal Data Sources page you may notice a sync error.

On the NFA Console <installdir>/reporter/logs/ProductSyncWSlog.log you could see:

Exception has been thrown by the target of an invocation.
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> NetQoS.ReporterAnalyzer.i18n.i18nException: Flatten failed on Interface ---> NetQoS.ReporterAnalyzer.i18n.i18nException: Flatten failed on Interface ---> NetQoS.ReporterAnalyzer.i18n.i18nArgumentException: Circular reference found in group relations.
at NetQoS.ReporterAnalyzer.Data.ReporterAnalyzer.ProductSync.PushedMembersDc.Flatten(String typeTable, String itemTypeName, UInt32 startTime, Boolean removeInheritance)

On the NetOps Portal <installdir>/PerformanceCenter/DM/logs/DMService.log

INFO | jvm 1 | AAAA/MM/DD hh:mm:ss | Data source DS(22) Network Flow [email protected] encountered an error while processing a sync request. The problem is in the data source, not CAPC. Check the logs for the data source to determine the cause of the problem. The following stack trace shows the context of the sync request:
INFO | jvm 1 | AAAA/MM/DD hh:mm:ss | javax.xml.ws.soap.SOAPFaultException: Server was unable to process request. ---> Exception has been thrown by the target of an invocation. ---> Flatten failed on Interface ---> Flatten failed on Interface ---> Circular reference found in group relations

Environment

NetOps Portal and NFA (all versions)

Cause

At some point the NFA Data Source was deleted from Performance Management then added back again. When a Data Source is deleted, Performance Management does not remove the Groups. Instead it moves them into a Data Source Orphan Groups list. Once NFA is added again and re-integrated with Performance Management the group was synchronized to Performance Management a second time. The end result is two instances of the Group. One is a working version of the group and one which is orphaned.

Resolution

To resolve this we will remove the Groups that are orphaned to ensure there is only one instance of the Group showing in Performance Management.

In the Manage Groups page go to the Orphan Groups list and delete all present.

All Groups -> Inventory -> Data Sources -> Data Source Orphan Groups

NOTE: The 'Data Source Orphan Groups' Group will only be visible in the Group section of the UI if the system does in fact have orphaned groups due to an originating data source having been removed. If it is not present, no cleanup via Group deletion is required.

In the Manage Groups page to the Data Sources User Defined Groups and delete all present.

All Groups -> Inventory -> Data Sources -> Network Flow Analysis -> User Defined Groups

If the problem persists after deleting 'Data Source Orphan Groups' and 'User Defined Groups' and waiting for a full NFA sync, please run through this procedure on the NetOps Portal server where MySQL resides:

mysqldump -uroot -p netqosportal > /tmp/netopsportalbackup.sql
mysql -uroot -p netqosportal -t -e "create table deleteme select sourceid,localid,itemtypename,min(updatedon) as updatedon from item_identifiers group by 1,2,3 having count(*) > 1;"
mysql -uroot -p netqosportal -t -e "delete from item_identifiers where (sourceid,localid,itemtypename,updatedon) in (select sourceid,localid,itemtypename,updatedon from deleteme);"
mysql -uroot -p netqosportal -t -e "drop table deleteme;"

When complete do a FULL sync with NFA. NFA will sync and complete then Global sync will begin. When Global sync completes NFA will do another sync and hopefully complete back to an Available status. If not, please start at step 1 again and run through the process again. It may need to be run a few times.