SDDC Manager UI fails to load with Java heap space "OutOfMemoryError"
search cancel

SDDC Manager UI fails to load with Java heap space "OutOfMemoryError"

book

Article ID: 437704

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • The SDDC Manager UI becomes unresponsive or fails to load across one or more Workload Domains (WLD).

  • Functionality is temporarily restored only after a manual restart of services or a reboot of the SDDC Manager VM.

  • /var/log/vmware/vcf/commonsvcs/commonsvcs.error reports

    Exception in thread "http-nio-127.0.0.1-7100-Acceptor" java.lang.OutOfMemoryError: Java heap space
            at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:64)
            at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:363)
            at org.apache.tomcat.util.net.SocketBufferHandler.<init>(SocketBufferHandler.java:56)
            at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:474)
            at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:70)
            at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:147)
            at java.base/java.lang.Thread.run(Thread.java:840)
    SLF4J(E): Failed toString() invocation on an object of type [java.util.LinkedHashMap]
    SLF4J(E): Reported exception:
    java.lang.OutOfMemoryError: Java heap space
    java.lang.OutOfMemoryError: Java heap space
    Exception in thread "http-nio-127.0.0.1-7100-Poller" java.lang.OutOfMemoryError: Java heap space 

 

Environment

SDDC Manager 9.0.x

 

Cause

commonsvcs service heap exhaustion (Out Of Memory) occurs when high API concurrency and resource-heavy operations exceed the default 2048 MB -Xmx limit.

Resolution

To resolve this issue, increase the maximum heap size for the commonsvcs service to 4096 MB.

Before proceeding, take a virtual machine snapshot of the SDDC Manager appliance.

  1. Log in to the SDDC Manager VM as vcf and switch to root.

  2. Verify the current heap usage by fetching the PID of commonsvcs service

    systemctl status commonsvcs

    Example: root@SDDC-Mgr [ /etc/vmware/vcf/domainmanager ]# systemctl status commonsvcs
    commonsvcs.service - VMware Cloud Foundation Platform Services
         Loaded: loaded (/etc/systemd/system/commonsvcs.service; enabled; preset: enabled)
         Active: active (running) since Fri YYYY-MM-DD HH:MM:SS UTC;
        Process: 1355 ExecStartPre=systemctl is-active commonsvcs-db.service (code=exited, status=0/SUCCESS)
        Process: 1360 ExecStartPost=/usr/bin/sh -c /usr/bin/sudo /bin/test -f /var/log/vmware/vcf/commonsvcs/commonsvcs.out && /bin/chown vcf_commonsvcs:vcf /var/log/vmware/vcf/commonsvcs/commonsvcs.out (code=exited, status=0/SUCCESS)
        Process: 1433 ExecStartPost=/usr/bin/sh -c /usr/bin/sudo /bin/test -f /var/log/vmware/vcf/commonsvcs/commonsvcs.err && /bin/chown vcf_commonsvcs:vcf /var/log/vmware/vcf/commonsvcs/commonsvcs.err (code=exited, status=0/SUCCESS)
       Main PID: 1359 (java)
          Tasks: 80 (limit: 19134)
         Memory: 1.9G

  3. Using the Main PID from Step 2, fetch the Xmx configuration

    Example: 
    ps -ef | grep 1359

    vcf_com+    1359       1  0 MMDD ?        HH:MM:SS /usr/lib/jvm/openjdk-java17-headless.x86_64/bin/java -Dorg.bouncycastle.fips.approved_only=true -Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true -XX:+UseParallelGC --add-opens java.base/java.lang.module=ALL-UNNAMED --add-opens java.base/jdk.internal.reflect=ALL-UNNAMED --add-opens java.base/jdk.internal.loader=ALL-UNNAMED --add-opens java.base/jdk.internal.module=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.ref=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.security.cert=ALL-UNNAMED -Djavax.xml.ws.spi.Provider=com.sun.xml.ws.spi.ProviderImpl -Xmx2048m -XX:ErrorFile=/var/log/vmware/vcf/commonsvcs/app-crash-logs/hs_err_pid_%p.log -Dspring.profiles.active=prod -Dapp.name=commonsvcs -cp /opt/vmware/vcf/fips/bc-fips-2.0.0.jar:/opt/vmware/vcf/fips/bctls-fips-2.0.19.jar:/opt/vmware/vcf/fips/bcutil-fips-2.0.3.jar:/opt/vmware/vcf/commonsvcs/lib/vcf-commonsvcs.jar org.springframework.boot.loader.JarLauncher

  4. Create a backup of the service configuration script:

    cp /opt/vmware/vcf/commonsvcs/bin/commonsvcs /home/vcf/commonsvcs_bak
     

  5. Modify the heap allocation to 4096 MB

    1.  vi /opt/vmware/vcf/commonsvcs/bin/commonsvcs

    2. Update the value from 2048M to 4096MB as below,

      #!/bin/bash
      # Copyright © 2024 Broadcom Inc. All rights reserved.

      # FIPS jars are added in classpath.This is required since java.security file is configured to use bouncy castle as a security provider and java needs these jars earlier before main() is called
      export SPRING_CONFIG_ADDITIONAL_LOCATION=file:/opt/vmware/vcf/commonsvcs/conf/

      exec /usr/lib/jvm/openjdk-java17-headless.x86_64/bin/java $JAVA_OPTS \
      -Xmx4096m \
      -XX:ErrorFile=/var/log/vmware/vcf/commonsvcs/app-crash-logs/hs_err_pid_%p.log \
      -Dspring.profiles.active=prod \
      -Dapp.name="commonsvcs" \
      -cp /opt/vmware/vcf/fips/bc-fips-2.0.0.jar:/opt/vmware/vcf/fips/bctls-fips-2.0.19.jar:/opt/vmware/vcf/fips/bcutil-fips-2.0.3.jar:/opt/vmware/vcf/commonsvcs/lib/vcf-commonsvcs.jar \
      org.springframework.boot.loader.JarLauncher 

  6. Restart the service to apply changes:

    systemctl restart commonsvcs

  7. Run the following command and ensure -Xmx4096m is present in the output:

    ps -ef | grep commonsvcs | grep Xmx

Powered by Wolken Software