As per JDK/JRE recommendation, you can allocate maximum of 80% of RAM to the resources running on system.
It also depends on the operating system process, number of services which you will deploy on DevTest VSE, number of test cases on Coordinator/Simulators ,etc.
Refer to Load and Performance Server Sizing section of the DevTest documentation for the release you are on.
Refer to Memory settings section of the DevTest documentation for the release you are on.
Note: A full list of JVM parameters is available from the Oracle website under Configuring the Default JVM and Java Arguments.