You will need to monitor the overall Java usage on the system and determine what peek usage is, then review the product documentation on increasing the JVM memory allocations:
For example of a system running Wildfly 15, if it is determined that IDM needs around 6gb of memory to function and avoid out of memory errors we would want to set something like the following for the JVM in the standalone.conf.bat:
-Xms8g -Xmx8g -XX:+UseG1GC -XX:ConcGCThreads=12 -XX:ParallelGCThreads=22
-XX:MaxGCPauseMillis=1000 -XX:InitiatingHeapOccupancyPercent=40
-XX:G1HeapWastePercent=2 -XX:G1ReservePercent=15
-XX:+UnlockExperimentalVMOptions -XX:G1OldCSetRegionThresholdPercent=15
-XX:G1MixedGCLiveThresholdPercent=90 -XX:G1NewSizePercent=20
-XX:G1MaxNewSizePercent=25
This will set the memory allocation to a total of 8gb to allow for some growth and to absorb unusually high activity, as well as tuning the garbage collection to better cleanup this larger allocation of memory.