SDDC Manager UI fails to load and commonsvcs service runs out of memory after repeated failed logins from an external IP address

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

The SDDC Manager UI fails to load.

The commonsvcs log /var/log/vmware/vcf/commonsvcs/commonsvcs.err contain the following:

Exception in thread "http-nio-127.0.0.1-7100-Acceptor" java.lang.OutOfMemoryError: Java heap space

 

Environment

SDDC Manager 5.x

 

Cause

This can happen when repeated failed logins from an external IP address cause the commonsvcs service to run out of memory.

Resolution

Verify that an IP address has been blocked in /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log:

YYYY-MM-DDTHH:mm:ss ERROR [common,xxxxxxxxxxxxxxxxxxxxxxxxxxxx,xxxx] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-62] [XXXXXX] CLIENT_IP_BLOCKED Client IP is blocked.
com.vmware.evo.sddc.identity.model.error.BlockedException: Client IP is blocked.

The offending IP address can be identified from repeated entries in /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs-activity.log:

YYYY-MM-DDTHH:mm:ss ERROR [common,xxxxxxxxxxxxxxxxxxxxxxxxxxxx,xxxx] [c.v.v.l.a.l.logger.ActivityLogger,http-nio-127.0.0.1-7100-exec-6] {"username":"[e-mail]","timestamp":"YYYY-MM-DDTHH:mm:ss","clientIP":"IP_ADDRESS","userAgent":"okhttp/2.7.5","api":"/v1/tokens","httpMethod":"POST","httpStatus":403,"operation":"Create Token Pair","remoteIP":"127.0.0.1"}

To resolve this issue, engage the vendor of the external system to reduce the frequency of the failed login attempts.