VSE is crashing after load test is performed

Products

• CA Application Test
• CA Continuous Application Insight (PathFinder)
• Service Virtualization

Issue/Introduction

The performance VSE service goes down unexpectedly after a load test is performed.

It was noticed the CPU and memory usage in the VSE server is very high after the load test and it does not go back to normal.

If VSE is not restarted, it crashes after some time.

Environment

All supported DevTest releases.
 

Cause

N/A

Resolution

Findings and Actions:
************************** 
In the local.properties file,  unlimited errors was set for VSE - lisa.vse.max.hard.errors=-1 
Performance VSE property was applied - lisa.vse.performance.enabled=true 
The vmoptions for VSE was set to 3G: 
-Xmx3072m  

During the load tests, there are no issues. No one is complaining about the load test not being completed.  
The problem is after the tests are ended the VSE crashes after some time.  

The load tests are being executed against data-driven virtual services and HTTPS is enabled with each of them.

We applied the following properties in the local.properties for VSE: 

lisa.pathfinder.on=false  
vse.socket.reuse=true  
lisa.net.timeout.ms=60000  
lisa.net.xstream=false  
lisa.dcm.maxExpandPerLoadChange=50  
lisa.jpm.MaxInstances=5000  

These properties come from the our documentation, refer to section "Sample Performance Benchmarks for Basic Service Virtualization" in the documentation of the DevTest release you are running.

Changed the log level to WARN for VSE and root category - this is the recommendation for load and performance environments. 

The issue was still happening. Recommended to apply the following property to your VSE properties file: 

lisa.vse.ssl.tcp.uses.nio=false 

After applying this property started seeing the following WARN messages: 

WARN com.itko.lisa.vse.sio.OldIOSession - A socket read timeout occurred. The read will be retried.  

Changed the log4j.rootCategory property to ERROR and stopped seeing the above WARN message. 

After facing some other issues with the environment, had to reinstall DevTest and recreate the data-driven virtual services. 

The issue was not happening when a 200/300 TPS load tests were performed. The last time the issue happened was after a 500 TPS test. 

Then able to run an 800 TPS load test and had no issues.