Seeing 100 % CPU and Memory spikes on VSE

Products

• Service Virtualization

Issue/Introduction

Seeing high CPU and Memory usage on a server where Registry, VSE and Portal are running and unable to list all virtual services in DevTest portal.

 

Environment

Release : 10.5

Component : CA Service Virtualization

Cause

Performance issue on a functional VSE with 1300VS's running caused the issue.

Resolution

 Making the below changes in the environment where Registry, VSE and  Portal are running resolved the issue. 

* Increased the memory as below.

Registry: Xms1024m & Xmx4096m

Portal: Xms1024m & Xmx4096m

VSE: Xms4096m & Xmx8192m

 

* Place the patch file "patch_DE502205_10.5.0_GA.jar" at LISA_HOME/lib/patches folder on Registry server

 

* Set the -Dlisa.amq.use,jmx=true  in VirtualServiceEnvironmentService.vmoptions

 

* Made the below changes to logging.properties file 

- Add  log4j.logger.com.itko.lisa.coordinator.EmbeddedWebServer=INFO

- Changed log4j.rootCategory=INFO,A1 ---> log4j.rootCategory=WARN,A1

- Changed log4j.logger.VSE=INFO, VSEAPP   --->  log4j.logger.VSE=WARN, VSEAPP

- Added log4j.logger.com.itko.lisa.coordinator.EmbeddedWebServer=INFO (After the issue is resolved this property can be commented)

- Added  log4j.logger.com.itko.lisa.coordinator.VirtualServiceEnvironmentImpl=INFO (After the issue is resolved this property can be commented)

 

* Set the below properties in local.properties

- lisa.pathfinder.on=false

- lisa.webserver.whitelist.hostsandpaths=0-255.0-255.0-255.0-255|/api/*,/lisa-virtualize-invoke/*,/acl/*,/api/*,/lisa-invoke/*,/lisa-pathfinder-invoke/*,/lisa-test-invoke/*,/devtest/*

- lisa.vse.pulseInterval=300   Added the following props to increased VS Status Pulse Interval from 30 seconds to 5 minutes. This is the periodic status update found in vse log file. Since Customer environment has 1300+ VS, it is better to increase the duration of printing this diagnostic information so that there is less load on VSE Server

 

 

- Added following properties to optimize the performance of VSE as a whole. Generally these are recommended only for Performance VSE but since we see 15000+ Live Threads for the 1300 VS's at normal load, these props are recommended:

vse.socket.reuse=true
lisa.net.timeout.ms=60000
lisa.net.xstream=false
activemq.msgCompression=true
lisa.jpm.MaxInstances=5000
lisa.jpm.unlimited=false
lisa.overloadThreshold=30000
lisa.eventPool.maxQueueSize=65535
lisa.eclipselink.cache.timeout.ms=3600000
lisa.CycleExecHistory.buffer.size=2