Install program setup.bin hangs java.lang.OutOfMemoryError: Java heap space

Products

• Workload Automation Agent

Issue/Introduction

Getting below error when trying to install esp workload automation agent on linux redhat R7 system.  setup.bin just hangs.  I noticed that I do get a file created in /tmp lax-7133461816657573380-err.tmp.  

Unable to invoke the method:
    EditVariableTable.setValue(String)
    Some installer configuration information may be lost
java.lang.OutOfMemoryError: Java heap space
        at java.lang.Class$MethodArray.<init>(Class.java:2726)
        at java.lang.Class$MethodArray.<init>(Class.java:2719)
        at java.lang.Class.privateGetPublicMethods(Class.java:2909)
        at java.lang.Class.privateGetPublicMethods(Class.java:2911)
        at java.lang.Class.privateGetPublicMethods(Class.java:2917)
        at java.lang.Class.privateGetPublicMethods(Class.java:2917)
        at java.lang.Class.privateGetPublicMethods(Class.java:2917)
        at java.lang.Class.privateGetPublicMethods(Class.java:2917)
        at java.lang.Class.privateGetPublicMethods(Class.java:2917)
        at java.lang.Class.getMethods(Class.java:1615)
        at Flexeraaok.az(Unknown Source)
        at Flexeraaok.as(Unknown Source)
        at Flexeraaok.ax(Unknown Source)
        at Flexeraaok.as(Unknown Source)
        at Flexeraaok.ax(Unknown Source)
        at Flexeraaok.as(Unknown Source)
        at Flexeraaok.as(Unknown Source)
        at Flexeraaok.as(Unknown Source)
        at Flexeraaok.ar(Unknown Source)
        at Flexeraaok.am(Unknown Source)
        at Flexeraaok.ak(Unknown Source)
        at Flexeraaok$1.aa(Unknown Source)

Environment

Release : 12.0

Component : CA Workload Automation System Agent

Cause

Insufficient memory for the installer.

Resolution

Run the installer with JVM heap size options provided there is enough memory on the server.

setup.bin -jvmxms <initial heap size> -jvmxmx <maximum heap size> 

For example:
# ./setup.bin -jvmxms 2G -jvmxmx 4G