Performance Center web UI is very slow, running into timeouts

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

After an upgrade of DX NetOps Performance Management (PM) from r3.7.18 to r20.2.8 several users are reporting the Performance Center (PC) web UI Portal is very slow.

Users are running into timeout issues with PC.

All parts of the PC web UI Portal including administration pages, Reports and Dashboards are slow.

Seeing this message a lot in the PCService.log file in (default path) /opt/CA/PerformanceCenter/PC/logs.

INFO   | jvm 1    | 2021/06/15 12:54:02 | WARN  | qtp118966853-23457       | 2021-06-15 12:54:02,151 | com.ca.im.portal.api.common.services.impl.database.DBPrioritizedProperties
INFO   | jvm 1    | 2021/06/15 12:54:02 |       | Could not obtain cacheUpdateLock within the timeout specified - cache was not updated.
Too many things using DBPrioritizedProperties and tell it to update cache, but only 1 can.  Rest will just use the new cached values after timeout.

Environment

All supported DX NetOps Performance Management releases

Cause

The error message indicates that there are many threads that are waiting for the DBPrioritizedProperties cache reload or a 5 second timeout to be reached.

If it is slow to load the netqosportal.performance_center_properties table values from the PC MySql netqosportal database, it slows down everything else trying to use that cache when the cache has expired and needs to be reloaded.

Resolution

The PrioritizedProperties.CacheSeconds is set to use a 60 second default cache timeout.

It is coded to use that default out of the box. To change that value we add it as a new entry to the (default path) /opt/CA/PerformanceCenter/PC/webapps/pc/WEB-INF/cfg/portal.console.properties file.

When setting the value we use a value representing a number of seconds. For example if we set it to a 3 minute timeout, we'd set it like this using 180 seconds.

• PrioritizedProperties.CacheSeconds=180

This will make it so the performance_center_properties table is only read every 3 minutes, which is fine since these properties don't, or shouldn't, be changing frequently.

If 180 seconds doesn't help, try raising it to 300 seconds (5 minutes).

After each time that value is changed, the PC Console service will need to be restarted to have it read the new value. To do this:

• RH 6.x releases:
  • service caperfcenter_console stop
  • service caperfcenter_console start
• RH 7.x releases:
  • systemctl stop caperfcenter_console
  • systemctl start caperfcenter_console