SDDC Manager UI will not load when Aria Operations is configured to connect to VCF with invalid credentials and 403 errors are found in commonsvcs activity log.

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• Restarting the SDDC Manager appliance or the Common Services service will restore the UI temporarily but the SDDC manager UI will again fail to load after some time. A monitoring system such as Aria Operations is configured to connect to VCF. The vcf-commonsvcs-activity logs are rolling over quickly and large in size.
• The commonsvcs.err log will show entries similar to: 

  java.lang.OutOfMemoryError: Java heap space

• The vcd-commonsvcs log will show entries similar to:

  status=500, error=Internal Server Error, exception=java.lang.OutOfMemoryError, trace=java.lang.OutOfMemoryError: Java heap space.

  ERROR "username":"<[email protected]>","timestamp":"<datetime>","clientIP":<ipadress>"  ... "api":"/v1/tokens","httpMethod":"POST","httpStatus":403,

• Reviewing the vcf-commonsvcs-activity log will show entries similar to occurring many times a second

Cause

The misconfigured service account rapidly querying the SDDC manager may eventually cause the Common Services service to run out of memory in Java heap.

Resolution

1. Review the vcf-commonsvcs-activity log to find the clientIP and username generating the traffic.
2. Update the system associated with the cilentIP to use valid credentials.
3. Restart SDDC Manager.
   

   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh