Receiving a 403 response code with an error code CLIENT_IP_BLOCKED when signing into SDDC Manager

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

Customers may receive a 403 response code with an error code CLIENT_IP_BLOCKED when signing into SDDC Manager using an account that has had 10 failed login attempts. Too many failed login attempts can lead to memory exhaustion in the java heap used by the commonsvcs service, causing the service to crash.
 
For the failed login attempts, vcf-commonsvcs.log will show errors similar to the following:
 

Line 262212: com.vmware.evo.sddc.identity.model.error.UnauthorizeException: User is not authorized
Line 2356048: 2023-10-17T22:45:06.684+0000 ERROR [common,af4058d4e458afdb,bc17] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-89301] [9M16I] IDENTITY_UNAUTHORIZED_ENTITY User is not authorized

 
Once the IP is blocked, vcf-commonsvcs.log will show errors similar to the following:
 

Line 2373963: 2023-10-17T23:00:07.294+0000 ERROR [common,b4389a784a0c53af,37e3] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-89300] [L3FJ18] CLIENT_IP_BLOCKED Client IP is blocked.
 Line 2373964: com.vmware.evo.sddc.identity.model.error.BlockedException: Client IP is blocked.
 Line 2374074: 2023-10-17T23:00:10.306+0000 ERROR [common,fa3a06520fc1f5f8,0651] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-89301] [I1KKPU] CLIENT_IP_BLOCKED Client IP is blocked.

 

Checking nginx-access.log we see the following from example IP 192.0.2.1: 

 192.0.2.1 - - [09/Dec/2024:14:01:19 +0000] "POST /v1/tokens HTTP/1.1" 403 119 "-" "okhttp/2.7.5" "-" 0.002 0.002
 192.0.2.1 - - [09/Dec/2024:14:01:19 +0000] "POST /v1/tokens HTTP/1.1" 403 118 "-" "okhttp/2.7.5" "-" 0.004 0.004
 192.0.2.1 - - [09/Dec/2024:14:01:19 +0000] "POST /v1/tokens HTTP/1.1" 403 119 "-" "okhttp/2.7.5" "-" 0.002 0.003
 192.0.2.1 - - [09/Dec/2024:14:01:19 +0000] "POST /v1/tokens HTTP/1.1" 403 119 "-" "okhttp/2.7.5" "-" 0.003 0.003
 192.0.2.1- - [09/Dec/2024:14:01:19 +0000] "POST /v1/tokens HTTP/1.1" 403 119 "-" "okhttp/2.7.5" "-" 0.003 0.003

Environment

VMware Cloud Foundation 4.x 

VMware Cloud Foundation 5.x

Cause

If a user tries to log in with invalid credentials 10 or more times, the client IP address will be blocked for 24 hours. 
The blocked IP will not be stored in a file, but rather in the in-memory cache.
 

Resolution

This is an expected behavior intended to protect the system from brute-force attacks.

Resolution: 

From the nginx.log, take a look at the IP that the 403 response code is coming from. This could be any component (Example: Aria Operations) using [email protected] credentials locking out due to a mismatch in passwords.

Workaround:

Restart the commonsvcs service using the following command:

systemctl restart commonsvcs