Dashboards taking several mins to load, overall slowness in NetOps

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

Dashboards not loading

Overall slowness

Overall slowness since upgrading to 21.2.1 

Overall slowness since upgrading to 21.2.2

Overall slowness since upgrading to 21.2.3

Performance issues since upgrading to 21.2.1

Performance issues since upgrading to 21.2.2

Performance issues since upgrading to 21.2.3 

 

Environment

Release : 21.2

Component : CA Performance Center

Cause

In 21.2.1, authentication between Performance Center and Data Aggregator was added for report API.  It introduced a connection pool from Device Manager to DA. The default connections for the pool was just determined to be 2.  So only 2 DA report calls could be made simultaneously.

Resolution

This issue is fixed in 21.2.4

Additional Information

To verify if you are seeing this issue you can enable debug and then check the DMservice.log for the number of connections and "Unable to get a connection, waiting"  messages. 

1. Enable debug.
 Go to the URL http://<CAPC_SYSTEM>:8181/pc/center/admin/debug/servers/dm/admin/config/logs?listing=all. Search for the category "org.apache.commons.httpclient.MultiThreadedHttpConnectionManager" and set the Level to debug and click the update button near the top of the page. 

2. Run one or more dashboards. 

3. Check the /opt/CA/PerformanceCenter/DM/logs/DMService.log for connections. If you only see 2 connection being used and "Unable to get a connection, waiting" you are experiencing this issue and need to upgrade to 21.2.4. 

[[email protected] logs]# tail -f DMService.log | grep connection
      | Getting free connection, hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Freeing connection, hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Getting free connection, hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Allocating new connection, hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Unable to get a connection, waiting..., hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Unable to get a connection, waiting..., hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Freeing connection, hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Freeing connection, hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Getting free connection, hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Getting free connection, hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Freeing connection, hostConfig=HostConfiguration[host=http://daHostname:8581]
      | Freeing connection, hostConfig=HostConfiguration[host=http://daHostname:8581]