Cpu utlization in CDD production DB server is 100%

Products

• Continuous Delivery Director SAAS

Issue/Introduction

Hello Team,

We are experiencing unusual utilization of CPU spike to 100% in last two days for the production DB server of CDD causing the CDD service to down. Kindly provide us support on replicating the cause of the problem and to make sure it not happens in future.

Cause

Root Cause

 

1)  CDD 6.9.0.1 is not supported anymore.

     We strongly recommend upgrading to the latest available CDD release 7.3.0.3

 

2) The top 12 longest REST API calls are coming from CDD Dashboard    

 

Please avoid using the CDD dashboard at CDD 7.1 as it is deprecated.

     

3) During the above time period (2020-09-17 22:26  until 2020-09-25 13:59 ) - there are 2,089 REST calls that are coming from CDD Dashboard.

    The duration of most of these calls is below 1 second.

    The duration of 1814 of these calls is below 7 seconds.

    The duration of 12 Dashboard calls is above 14 minutes (!).

 

4) We strongly recommend running the production MySQL database on a dedicated 32GB RAM machine that is not running any other processes ( such as Java processes ).

     Could you please consider moving the Java processes from the production MySQL database machine to a different machine.

 

5) Could you please reduce/stop the usage of the CDD Dashboard.

•   CDD Dashboard was deprecated feature and removed from the latest CDD release and not supported anymore. 
•   At CDD 7,1, the polling period of the CDD dashboard was reduced to once in every 10 minutes. 
•   CDD dashboard would be replaced by dedicated reports and time series at one of the coming CDD versions. The dashboard is completely removed at CDD 8.0.

Environment

Release : 7.1

Component : CONTINUOUS DELIVERY DIRECTOR

Resolution

Recommendation: We strongly recommend upgrading to the latest available CDD release 7.3.0.3

Additional Information

1. At the next time you would experience high memory consumption on the database machine, could you please share the top memory consuming SQL statements.
2. You may increase the number/size of CDD server log files to 50 log files of 50MB each.

settings.properties
======================================
cdd.log.cdd-server.maxSize = 50MB
cdd.log.cdd-server.maxIndex = 50
======================================