SDDC Manager UI is slow or not responding

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

Unable to login to SDDC Manager UI - issue could be intermittent
SDDC Manager UI is slow after user login.

Environment

VMware Cloud Foundation 3.x

VMware Cloud Foundation 4.x

Cause

Operations related to password or credential consume resources on SDDC Manager Appliance.

You can see the following messages in vcf-commonsvcs.log, or SDDC Manager Appliance shows a high CPU usage via vSphere Client.

vcf-commonsvcs.log

YYYY-MM-DDTHH:MM:SS.SSS+0000 ERROR [common,0000000000000000,0000] [o.a.coyote.http11.Http11NioProtocol,http-nio-127.0.0.1-7100-exec-34094] Failed to complete processing of a request
java.lang.OutOfMemoryError: Java heap space
YYYY-MM-DDTHH:MM:SS.SSS+0000 ERROR [common,0000000000000000,0000] [o.a.coyote.http11.Http11NioProtocol,http-nio-127.0.0.1-7100-exec-34096] Failed to complete processing of a request
java.lang.OutOfMemoryError: Java heap space
YYYY-MM-DDTHH:MM:SS.SSS+0000 ERROR [common,0000000000000000,0000] [o.a.coyote.http11.Http11NioProtocol,http-nio-127.0.0.1-7100-exec-34091] Failed to complete processing of a request
java.lang.OutOfMemoryError: Java heap space
YYYY-MM-DDTHH:MM:SS.SSS+0000 ERROR [common,cfa7a58660a9c894,c21c] [o.h.e.jdbc.spi.SqlExceptionHelper,http-nio-127.0.0.1-7100-exec-34079] HikariPool-1 - Connection is not available, request timed out after 71607ms.
YYYY-MM-DDTHH:MM:SS.SSS+0000 ERROR [common,cf3e7eac019a4d7e,fd76] [o.h.e.jdbc.spi.SqlExceptionHelper,http-nio-127.0.0.1-7100-exec-34078] HikariPool-1 - Connection is not available, request timed out after 41188ms.
YYYY-MM-DDTHH:MM:SS.SSS+0000 INFO [common,015965f86415609d,6029] [o.s.core.annotation.MergedAnnotation,http-nio-127.0.0.1-7100-exec-34090] Failed to introspect annotations on public java.util.Collection com.vmware.evo.sddc.inventory.rest.api.controller.ResourceInventoryController.getSddcManagerControllers(java.lang.String): java.lang.OutOfMemoryError: Java heap space
YYYY-MM-DDTHH:MM:SS.SSS+0000 ERROR [common,0000000000000000,0000] [o.apache.tomcat.util.net.NioEndpoint,http-nio-127.0.0.1-7100-exec-34096] Error running socket processor
java.lang.NullPointerException: null

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Resolution

This issue is resolved in VMware Cloud Foundation 5.0.

Workaround:

To resolve the issue follow the steps below:

Note: Take a thread dump before restarting service if the issue still occurs in the fixed version.

File a support request and upload SDDC Manager log and the above thread dump.

Log in to the SDDC Manager appliance by using a Secure Shell (SSH) client as vcf.
Elevate to root permissions by running the su - command.
Restart the commonsvcs service by running this command.
# systemctl restart commonsvcs
To verify the commonsvcs service has started, run this command.
# curl localhost/commonsvcs/about
Output should look similar to:

{"id":"<UUID>","name":"COMMON_SERVICES","version":"4.5.1-vcf4510RELEASE-21682124","status":"ACTIVE","serviceUrl":"http://localhost/commonsvcs","description":"Platform services"
Refresh the Web browser window that shows the SDDC Manager UI.

If SDDC Manager UI doesn't still work, then try to reboot SDDC Manager Appliance.

Additional Information

[Appendix A] Take a thread dump

Find PID for commonsvcs. PID is outputted after "Main PID:"
# systemctl status commonsvcs

Take a thread dump.
# jstack -e -l <PID> > commonsvcs_threads.txt

ex.) If PID=2370, you can run this:
# jstack -e -l 2370 > commonsvcs_threads.txt

[Appendix B] Collect Logs for Your VMware Cloud Foundation System
Administration Guide