Delete items and break the MP<->Collection association

Products

DX NetOps CA Performance Management - Usage and Administration

Issue/Introduction

After associating QOS monitoring profile to all manageable devices, the DA becomes bogged down and unusable

DA fails sync

Environment

Performance Management - All Versions

Cause

Millions of facets have been added to the database, and Monitoring Profile <-> Collection association

Resolution

Manually remove items and Monitoring Profile <-> Collection association, In the Vertica database:

1. Get a listing of facet counts, as follows
2. Login to vsql as your database user; default is dauser; vsql -U dauser
3. Get a facet count by running the following select statement, based on QoS items; in this case

[DB_Name]=> select facet_qname,count(*) from v_item_facet where facet_qname like '%}QoS%' group by 1 order by 2 desc;
facet_qname | count
-----------------------------------------------+---------
{http://im.ca.com/inventory}QoSContract | 4793805
{http://im.ca.com/inventory}QoSClassMap | 1495376
{http://im.ca.com/inventory}QoSQueuing | 934323
{http://im.ca.com/inventory}QoSPolicer | 47984
{http://im.ca.com/inventory}QoSRED | 7936
{http://im.ca.com/inventory}QoSTrafficShaping | 443
(6 rows)

[DB_Name]=> select facet_qname, count(*) from v_item_facet group by 1 order by 2 desc;
facet_qname | count
---------------------------------------------------------------------------------------+---------
{http://im.ca.com/inventory}Pollable | 8360872
{http://im.ca.com/inventory}DeviceComponent | 8018234
{http://im.ca.com/core}Syncable | 7961397
{http://im.ca.com/inventory}Hierarchy | 7279867
{http://im.ca.com/inventory}QoSContract | 4793805
{http://im.ca.com/core}GroupHasMember | 2932771
{http://im.ca.com/inventory}QoSClassMap | 1495376
{http://im.ca.com/inventory}QoSQueuing | 934323
{http://im.ca.com/inventory}Port | 597996
{http://im.ca.com/sdn}SDN | 566558
{http://im.ca.com/pollingconfig}PollingConfig | 474229
{http://im.ca.com/core}IPDomainMember | 411071
{http://im.ca.com/sdn}VNAPollConfig | 391963
{http://im.ca.com/core}Retired | 336617
{http://im.ca.com/inventory}Tunnel | 284709
{http://im.ca.com/inventory}TopLink | 283073
{http://im.ca.com/normalizer}NormalizedSDNTunnelInfo | 257906
{http://im.ca.com/inventory}DiscoveryInfo | 195337
{http://im.ca.com/sdn}SDNTemporaryRelation | 167757
{http://im.ca.com/sdn}SDNSLAPath | 106705
{http://im.ca.com/sdn}SDNRelation | 95069

d. Stop your DA before proceeding to the following steps;

systemctl stop dadaemon (for single DA)

maintenance dadaemon ( for fault tolerant DAs, run on both DAs )

e. login to vsql as your database user; dauser

f. create table deleteitems(item_id int);

g. insert into deleteitems select distinct item_id from v_item_facet where facet_qname like '%}QoS%';

h. delete from item where exists (select null from deleteitems d where d.item_id=item.item_id);

i. Validate should be 0, for the following select statement
select facet_qname,count(*) from v_item_facet where facet_qname like '%}QoS%' group by 1 order by 2 desc;

j. commit;

2. Run the following to start your Data Aggregator

systemctl start dadaemon ( on single DA )

dadaemon activate ( on fault tolerant DAs )

3. Log into the NetOps portal, and remove the relationship from the QoS Monitoring Profile <> All Monitored Devices collection

4. On the NetOps Portal MySql database, do the following to cleanup items there.

Login to MySql: /opt/CA/MySql/bin/mysql -uroot -p
provide the password and type "use netqosportal"
Run the following

select itemsubtypename,count(*) from dst_component where itemsubtypename like 'qos%' group by 1;

mysql> select itemsubtypename,count(*) from dst_component where itemsubtypename like 'qos%' group by 1;
+-------------------+----------+
| itemsubtypename | count(*) |
+-------------------+----------+
| qosclassmap | 1316875 |
| qoscontract | 4613829 |
| qospolicer | 42776 |
| qosqueuing | 889138 |
| qosred | 7872 |
| qostrafficshaping | 442 |
+-------------------+----------+
6 rows in set (37.74 sec)

d. Run the following commands to set all qos* components to be marked for deletion from Portal.

update ds_items set UpdatedOn=0 where itemtypename='component' and itemsubtypename like 'qos%';
update dst_component set UpdatedOn=0 where itemsubtypename like 'qos%';

Then once the commands are done, the NEXT global sync to run will remove them from the NetOps Portal MySql DB.

Once the Portal Items are down to 0 again, check the following for any remaining QoS associations

http://<dahost>:8581/debug/dispatcher/container/com.ca.im.aggregator.discovery/bean/CMPChangDetectionScheduler

5. If there are remaining QoS associations in the CMP scheduler bean, you'll need to trigger a recalculation of the CMP for the associated devices and stop the QoS item discovery from continuing

Copy the Reachability MP. And choose like 1 hr poll rate. 24 hr CD. Save.
Associate new MP to the All Manageable Devices. Wait 15 mins.
Check the CMP scheduler bean to see if the QoS MF are removed from the CMP scheduler bean for all devices.

http://dahost:8581/debug/dispatcher/container/com.ca.im.aggregator.discovery/bean/CMPChangDetectionScheduler
If all cleaned up, Unassociate All Manageable Devices from new Monitoring Profile.
Delete new Monitoring Profile