APM Command Center Server stops working - Persistent store is Full
Article ID: 109925
Updated On:
Products
CA Application Performance Management Agent (APM / Wily / Introscope)
INTROSCOPE
Issue/Introduction
APM Command Center Server (ACC) server stops working properly.
In the apmccsrv.log the below message is reported:
2018-07-31 14:19:37.773 INFO 18846 [qtp352138147-127854] --- org.apache.activemq.broker.region.Queue : Usage(Main:store:queue://com.ca.apm.acc.cfgsrv.ctrl.377DC56F3E8379A0BD509EFAB98D66750445ED89F5A691D269BE988993E7650E:store) percentUsage=99%, usage=1099791482, limit=1073741824, percentUsageMinDelta=1%;Parent:Usage(Main:store) percentUsage=102%, usage=1099791482, limit=1073741824, percentUsageMinDelta=1%: Persistent store is Full, 100% of 1073741824. Stopping producer (ID:DCDEFRDLX006VD-58988-1530018175911-2:10:1:1) to prevent flooding queue://com.ca.apm.acc.cfgsrv.ctrl.377DC56F3E8379A0BD509EFAB98D66750445ED89F5A691D269BE988993E7650E. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 78804s)
In the apmccsrv.log the below message is reported:
2018-07-31 14:19:37.773 INFO 18846 [qtp352138147-127854] --- org.apache.activemq.broker.region.Queue : Usage(Main:store:queue://com.ca.apm.acc.cfgsrv.ctrl.377DC56F3E8379A0BD509EFAB98D66750445ED89F5A691D269BE988993E7650E:store) percentUsage=99%, usage=1099791482, limit=1073741824, percentUsageMinDelta=1%;Parent:Usage(Main:store) percentUsage=102%, usage=1099791482, limit=1073741824, percentUsageMinDelta=1%: Persistent store is Full, 100% of 1073741824. Stopping producer (ID:DCDEFRDLX006VD-58988-1530018175911-2:10:1:1) to prevent flooding queue://com.ca.apm.acc.cfgsrv.ctrl.377DC56F3E8379A0BD509EFAB98D66750445ED89F5A691D269BE988993E7650E. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 78804s)
Environment
ACC (any version).
Cause
The problem is due to the amount of agents (and probably controllers), activeMQ broker runs out of persistent store for messages. It because of a burst of messages at one time (e.g. many agents or controllers connecting) or it may not keep up with arriving number of messages.
Resolution
1.- Stop ACC server
2.- Adjust the ACC configuration settings in the ACC_HOME/config/apmccsrv.properties by adding the below hidden properties:
a) Increase the size of the persistent store. Default size limit is 1GB, try doubling the size to 2GB.
com.ca.apm.acc.broker.storeUsageLimit=2147483648
b) Increase the size of temp store. Default size of 100MB, try setting this to some higher value, e.g. 1GB
com.ca.apm.acc.broker.tempUsageLimit=1073741824
If the above doesn’t' help, you can try to clear all the data and let all the agent to reconnect from scratch again
Backup and delete the activemq-data directory in ACC folder.
3.- Start ACC server
2.- Adjust the ACC configuration settings in the ACC_HOME/config/apmccsrv.properties by adding the below hidden properties:
a) Increase the size of the persistent store. Default size limit is 1GB, try doubling the size to 2GB.
com.ca.apm.acc.broker.storeUsageLimit=2147483648
b) Increase the size of temp store. Default size of 100MB, try setting this to some higher value, e.g. 1GB
com.ca.apm.acc.broker.tempUsageLimit=1073741824
If the above doesn’t' help, you can try to clear all the data and let all the agent to reconnect from scratch again
Backup and delete the activemq-data directory in ACC folder.
3.- Start ACC server
Feedback
Yes
No
Powered by
