NetOps Performance Manager Event Manager service logs filling up disk

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

I'm having trouble determining what is causing the directory to fill up with 100gb of data every few days.  The wrapper log files are becoming problematic.

The DX NetOps Performance Manager Event Manager service logs are filling up the servers disk space. The offending logs are in the (default path shown) /opt/CA/PerformanceManager/EM/logs directory.

These are the errors seen filling the logs. They are very large errors with more than 65K characters per message. They're seen generally with every attempted Event Manager Data Source Synchronization cycle.

Error sample showing the start of the error.

INFO   | jvm 1    | 2022/12/18 00:06:44 | ERROR | pool-6-thread-762        | 2022-12-18 00:06:44,015 | com.ca.im.portal.em2.cache.GroupPathUtils
INFO   | jvm 1    | 2022/12/18 00:06:45 |       | org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [INSERT INTO item_grouppath_work(ItemID, GroupPath
) VALUES (107,'#1#37474207#5#6#7#37180478#12399095#107#12#27340749#92276396#91511961#58473518#92276563#53#23440210#33145614#74843873#1093835#15017182#16815227#88455998#12452547#'),(109,'#12648641#18358552#1#12603585#12603587#5#12603589#6#12619974#12648647#83977489#83977488#83977490#33173998#16625896#83977494#83977497#16625895#17727764#69751858#83977499#83977500#13123799#18170122#16548062#23499072#12620010#12620012#12620016#18055459#70980618#12620018#70980617#62120832#12620020#12620022#22511982#19955980#19955981#22864225#19955982#1265677...

Environment

All supported DX NetOps Performance Management releases

Cause

Unexpectedly large MySQL packet size, 128MB packet for 5000 rows being sent, while max_allowed_packet configuration in MySQL is too low.

 

Resolution

Defect DE553614 provides code change that resolves this problem. The changes are scheduled for inclusion in the 22.2.5 release. Upgrade to that release when available to resolve this problem.

To limit the log growth pending a solution we can modify the log configuration. The following change will lower the number of wrapper logs being stored. Dropping default 5 logs max to 1 log max provides an 80% drop for the rate logs will accumulate. To make this change take these steps. Note this will be reverted to default during the upgrade that brings new code to resolve this.

1. Go to (default paths shown) /opt/CA/PerformanceCenter/EM/conf.
2. Copy the wrapper.conf to create a backup copy with this command:
   • cp wrapper.conf /opt/CA/PerformanceCenter/EM/conf/wrapper.conf.orig
3. Edit the remaining wrapper.conf file.
4. Change the default value of 5 in wrapper.logfile.maxfiles. Set it to 1.
   • This line:
     • wrapper.logfile.maxfiles=5
   • Becomes:
     • wrapper.logfile.maxfiles=1
5. Save the changes to the file.
6. Restart the Event Manager service to implement the change.
   • Stop the service:
     • systemctl stop caperfcenter_eventmanager
   • Start the service:
     • systemctl start caperfcenter_eventmanager

Additional Information

Section from /opt/CA/PerformanceCenter/EM/conf/wrapper.conf file where edit is required:

# Maximum number of rolled log files which will be allowed before old
#  files are deleted.  The default value of 0 implies no limit.
wrapper.logfile.maxfiles=5