Global Synchronization in DX NetOps Portal runs for hours.

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

Global sync for CAPC is currently taking 3+ hours, need to extend the timeout/delay between starting the next global sync as we're getting multiple going at a time. I believe we're currently sitting at 2 hours.

DX NetOps Performance Management PC Portal web server Global Synchronization cycles should normally complete within 5-10 minutes. Sometimes it may run longer when addressing large change sets.

It should never run for many hours for multiple cycles one after the other. In this case the Global Synchronization cycles would take 3+ hours to complete. All runs would take this long.

It would cause Dashboard and Report failures due to query timeouts due to MySql being too busy to respond while Global Synchronization was running and slowing things down.

Environment

All supported DX Performance Management releases

Cause

There was some tuning of MySql that improved the time frames by 60-90 minutes.

The remaining time frame was due to excessive item counts in the PC Portal MySql netqosportal DB. Primarily Process CPU Usage items.

Resolution

Some time was reduced through MySql configuration tuning performed with engineering assistance and input.

The primary solution was identifying and removing more than 5.2 million processcpuusage items. They are known to cause excess item count issues. Everytime processes change, we remove the item and create a new one. Generates excessive numbers of old/current items in the PC DB. We believe they were generated with the Metric Family was tied to all devices. Then quickly removed. The mass change failed to be properly unsynced from PC Portal inventory.

Once removed it lowered Global Synchronization cycle times to less than 10 minutes.

Additional Information

What do Global Synchronization cycle time frames look like historically and currently? Gather the following output which will show the cycle time frames for the last few days.

• This uses the default path. Change it match yours if different.
• It creates a file in /tmp named GlobalSyncTimes.out. Change the path if /tmp isn't a good choice.
• Review the times to see how it's been running.
  • The "from_unixtime(starttime) | mins" column will show the number of 'minutes.seconds' it took for the cycle to run total.
  • The cycle start time is shown in the "from_unixtime(synctime)" column.
• Run this command. Enter the password when prompted.
  • /opt/CA/MySql/bin/mysql -uroot -p -e "select from_unixtime(synctime), stagename, from_unixtime(starttime), (endtime-starttime)/60 as mins from netqosportal.global_sync_log where stagename='GlobalSync' order by from_unixtime(starttime) asc;' >> /tmp/GlobalSyncTimes.out