QA Environment is a distributed deployment of DevTest 10.7.0 on Windows Server 2019 connecting to an Oracle19c external database.
We recently had a situation where the Enterprise Dashboard service ran out of memory and crashed. We increased the memory to 4 GB and restarted all the service. During the troubleshooting process, we noticed the Access Event Log has a high row count, over 55 million rows. We are not sure if this is normal, or if this is a potential issue.
Access event log(from Enterprise dashboard Database) and Dradiscache(from registry database)
We are retaining metric data for 120 days. See the following parameters in the local.properties file:
perfmgr.rvwiz.whatrpt.expireTimer=120d
lisa.vse.metrics.delete.age=120d
Looking at the registry.log, we see "sendAccessEventToDradis" recorded approvingly every 20+ milliseconds, and the CPU for the Registry server is at about 50% utilization.
Also in the enterprisedashboard.log we see "DEBUG com.ca.dradis.util.ObjectStore - size of object store is - 64078" and would like to know if this is a normal queue size.
Release : 10.7.0
Product Defect.
Open a support case and refer to defect DE555170.
The Enterprise Dashboard log revealed that access events are getting out of turn. These session events are coming in very quickly. Basically, they are a pair of events; one to create a session and another to destroy the session. In short, the event to destroy the session is getting processed before the create session event, which is causing the issue, which is also causing a build-up of memory.
Engineering has developed a patch to address this issue. The patch changes the method used to process the events in the ObjectStore.