Enterprise Dashboard table Access Event Log has over 55 million rows

Products

• Service Virtualization

Issue/Introduction

QA Environment is a distributed deployment of DevTest 10.7.0 on Windows Server 2019 connecting to an Oracle19c external database.

• IAM and Registry on one server - Windows Server 2019 server with 4 CPU cores (quad 260 Ghz) and 16 GB of RAM
  • RegistryService.vmoptions = -Xmx5120m
  • PortalService.vmoptions =-Xmx4096m
  • Patches:
    • patch_DE532649_DE537095_+DE543873_10.7.0_GA.jar
    • lisa-virtualize-invoke-10.7.0-DE550631.war
      
      
• Enterprise Dashboard and Portal on one server - Windows Server 2019 server with 4 CPU cores (quad 260 Ghz) and 16 GB of RAM
  
  • EnterpriseDashboardService.vmoptions originally was set to -Xmx2048m we increased this to -Xmx4096m.
    
    
• 4 VSEs all running on their own Server.- Windows Server 2019 server - 8 core with 32GB RAM.

We recently had a situation where the Enterprise Dashboard service ran out of memory and crashed. We increased the memory to 4 GB and restarted all the service. During the troubleshooting process, we noticed the Access Event Log has a high row count, over 55 million rows. We are not sure if this is normal, or if this is a potential issue. 

Access event log(from Enterprise dashboard Database) and Dradiscache(from registry database)

• Access Event log: 55896016
• Dradiscache: don't have data 0 rows.

We are retaining metric data for 120 days. See the following parameters in the local.properties file:

perfmgr.rvwiz.whatrpt.expireTimer=120d
lisa.vse.metrics.delete.age=120d

Looking at the registry.log, we see "sendAccessEventToDradis" recorded approvingly every 20+ milliseconds, and the CPU for the Registry server is at about 50% utilization. 

Also in the enterprisedashboard.log we see "DEBUG com.ca.dradis.util.ObjectStore - size of object store is - 64078" and would like to know if this is a normal queue size.

Environment

Release : 10.7.0

Cause

Product Defect.

Resolution

Open a support case and refer to defect DE555170.

Additional Information

The Enterprise Dashboard log revealed that access events are getting out of turn. These session events are coming in very quickly. Basically, they are a pair of events; one to create a session and another to destroy the session. In short, the event to destroy the session is getting processed before the create session event, which is causing the issue, which is also causing a build-up of memory.

Engineering has developed a patch to address this issue. The patch changes the method used to process the events in the ObjectStore.