ccsmact performance problem - all connections closed
search cancel

ccsmact performance problem - all connections closed

book

Article ID: 372498

calendar_today

Updated On:

Products

CA Client Automation - IT Client Manager CA Client Automation

Issue/Introduction

ccsmact plugin (Configuration and State Management agent controller) has performance problem. Configuration jobs and OSIM jobs are activated and processed slowly. ccsmactd.exe process is taking CPU and lot of memory.

In TRC_CSMACT*.log we could see following message :

200624-08:36:41.0728435L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 143 **
200624-08:42:55.0754794L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 222 **
200624-08:43:16.5383760L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 227 **
200624-08:49:34.3315709L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 270 **
...
200624-08:57:18.3810220L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 288 **
200624-08:57:50.3806385L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 290 **
200624-08:58:12.3020987L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 292 **
200624-08:58:40.0617507L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 295 **
...
200624-09:10:48.0905723L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 361 **
200624-09:25:03.0620689L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 461 **
200624-09:25:03.0621519L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000864|INFO   | - Run() m_nState=<6>
200624-09:25:03.0622234L|011232|00002be8|ccsmact |CFSMCAPI |CFSMCAPI   |000000|DETAIL | SmCloseEndPoint : API Entry
200624-09:25:03.0622756L|011232|00002be8|ccsmact |CFSMCAPI |CFSMCAPI   |000000|DETAIL | deleteEndpoint : set m_state = 2. Session closing - before critical section
200624-09:25:03.0623264L|011232|00002be8|ccsmact |CFSMCAPI |CFSMCAPI   |000000|DETAIL | SmiEndPoint::smShutdown : Endpoint CAI011232-00003
200624-09:25:03.0623961L|011232|00002be8|ccsmact |CFSMCAPI |CFSMCAPI   |000000|DETAIL | SmiSessionTable::closeAllSessions : Closing all open sessions. sessions count <1>

There are a lot of scalability servers (1600 or more).

The sessions and objects settings stored in memory for a scalability server is taking in ccsmact memory about 250KB.

With 400 MB ccsmact could store about 1600 scalability servers. If there are more than 1600 scalability servers, this problem could occur and max memory usage should be increased

Environment

Client Automation 14.5

Cause

Regularly, ccsmact checks its memory usage

Example :

200624-08:36:41.0728435L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 143 **
200624-08:42:55.0754794L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 222 **
200624-08:43:16.5383760L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 227 **

"Force job check 143" means that it is using 143 MB of memory

When memory usage exceeds 400 MB, ccsmact closes all the sessions opened for all scalability servers :

200624-09:25:03.0620689L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000993|NOTIFY | @ Run(): *** Force job check 461 **
200624-09:25:03.0621519L|011232|00002be8|ccsmact |csmtrace |csmact.cpp |000864|INFO   | - Run() m_nState=<6>
200624-09:25:03.0622234L|011232|00002be8|ccsmact |CFSMCAPI |CFSMCAPI   |000000|DETAIL | SmCloseEndPoint : API Entry
200624-09:25:03.0622756L|011232|00002be8|ccsmact |CFSMCAPI |CFSMCAPI   |000000|DETAIL | deleteEndpoint : set m_state = 2. Session closing - before critical section
200624-09:25:03.0623264L|011232|00002be8|ccsmact |CFSMCAPI |CFSMCAPI   |000000|DETAIL | SmiEndPoint::smShutdown : Endpoint CAI011232-00003
200624-09:25:03.0623961L|011232|00002be8|ccsmact |CFSMCAPI |CFSMCAPI   |000000|DETAIL | SmiSessionTable::closeAllSessions : Closing all open sessions. sessions count <1>

This is affecting the performance because ccsmact should reinitialize again every connection with all scalability servers and this is consuming a lot of time.

Resolution

  1. Execute following command on Domain Manager to increase the Max Memory Usage to 800 MB
    ccnfcmda -cmd SetParameterValue -ps itrm/manager/cc -pn MaxMemUsage -v 800
     
    You could check the value with this command: 
    ccnfcmda -cmd GetParameterValue -ps itrm/manager/cc -pn MaxMemUsage


  2. Then restart plugin ccsmact :

    caf stop ccsmact
    caf start ccsmact

Additional Information

In TRC_CSMACT*.log, we could see this :

170724-09:16:45.1688214L|010800|000026ac|ccsmact   |csmtrace        |csmact.cpp          |000614|NOTIFY | @ GetConfiguration(): *** Max memory:800 MB ***
Powered by Wolken Software