The performance VSE service goes down unexpectedly after a load test is performed.
It was noticed the CPU and memory usage in the VSE server is very high after the load test and it does not go back to normal.
If VSE is not restarted, it crashes after some time.
All supported DevTest releases.
Findings and Actions:
In the local.properties file, unlimited errors was set for VSE - lisa.vse.max.hard.errors=-1
Performance VSE property was applied - lisa.vse.performance.enabled=true
The vmoptions for VSE was set to 3G:
During the load tests, there are no issues. No one is complaining about the load test not being completed.
The problem is after the tests are ended the VSE crashes after some time.
The load tests are being executed against data-driven virtual services and HTTPS is enabled with each of them.
We applied the following properties in the local.properties for VSE:
These properties come from the our documentation, refer to section "Sample Performance Benchmarks for Basic Service Virtualization" in the documentation of the DevTest release you are running.
Changed the log level to WARN for VSE and root category - this is the recommendation for load and performance environments.
The issue was still happening. Recommended to apply the following property to your VSE properties file:
After applying this property started seeing the following WARN messages:
Changed the log4j.rootCategory property to ERROR and stopped seeing the above WARN message.
After facing some other issues with the environment, had to reinstall DevTest and recreate the data-driven virtual services.
The issue was not happening when a 200/300 TPS load tests were performed. The last time the issue happened was after a 500 TPS test.
Then able to run an 800 TPS load test and had no issues.