search cancel

Seeing 100 % CPU and Memory spikes on VSE

book

Article ID: 215367

calendar_today

Updated On:

Products

Service Virtualization

Issue/Introduction

Seeing high CPU and Memory usage on a server where Registry, VSE and Portal are running and unable to list all virtual services in DevTest portal.

 

Cause

Performance issue on a functional VSE with 1300VS's running caused the issue.

Environment

Release : 10.5

Component : CA Service Virtualization

Resolution

 Making the below changes in the environment where Registry, VSE and  Portal are running resolved the issue. 

* Increased the memory as below.
Registry: Xms1024m & Xmx4096m
Portal: Xms1024m & Xmx4096m
VSE: Xms4096m & Xmx8192m
 
* Place the patch file "patch_DE502205_10.5.0_GA.jar" at LISA_HOME/lib/patches folder on Registry server
 
* Set the -Dlisa.amq.use,jmx=true  in VirtualServiceEnvironmentService.vmoptions
 
* Made the below changes to logging.properties file 
- Add  log4j.logger.com.itko.lisa.coordinator.EmbeddedWebServer=INFO
- Changed log4j.rootCategory=INFO,A1 ---> log4j.rootCategory=WARN,A1
- Changed log4j.logger.VSE=INFO, VSEAPP   --->  log4j.logger.VSE=WARN, VSEAPP
- Added log4j.logger.com.itko.lisa.coordinator.EmbeddedWebServer=INFO (After the issue is resolved this property can be commented)
- Added  log4j.logger.com.itko.lisa.coordinator.VirtualServiceEnvironmentImpl=INFO (After the issue is resolved this property can be commented)
 
* Set the below properties in local.properties
- lisa.pathfinder.on=false
- lisa.webserver.whitelist.hostsandpaths=0-255.0-255.0-255.0-255|/api/*,/lisa-virtualize-invoke/*,/acl/*,/api/*,/lisa-invoke/*,/lisa-pathfinder-invoke/*,/lisa-test-invoke/*,/devtest/*
- lisa.vse.pulseInterval=300   Added the following props to increased VS Status Pulse Interval from 30 seconds to 5 minutes. This is the periodic status update found in vse log file. Since Customer environment has 1300+ VS, it is better to increase the duration of printing this diagnostic information so that there is less load on VSE Server
 
 
- Added following properties to optimize the performance of VSE as a whole. Generally these are recommended only for Performance VSE but since we see 15000+ Live Threads for the 1300 VS's at normal load, these props are recommended:
vse.socket.reuse=true
lisa.net.timeout.ms=60000
lisa.net.xstream=false
activemq.msgCompression=true
lisa.jpm.MaxInstances=5000
lisa.jpm.unlimited=false
lisa.overloadThreshold=30000
lisa.eventPool.maxQueueSize=65535
lisa.eclipselink.cache.timeout.ms=3600000
lisa.CycleExecHistory.buffer.size=2