High CPU utilization and Out of Memory messages in vIDM 3.3.x

Products

• VMware Aria Suite

Issue/Introduction

High CPU utilization and OutOfMemory error seen in horizon.log , gradually the vIDM system may become unresponsive.

In the /opt/vmware/horizon/workspace/logs/horizon.log file, you see entries similar to:

2020-04-16T03:34:01,512 WARN  (C3P0PooledConnectionPoolManager[identityToken->z8kflta8pplzokkop8au|76637162]-HelperThread-#2) [;;;] com.mchange.v2.async.ThreadPoolAsynchronousRunner - An Error forced the closing of Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta8pplzokkop8au|76637162]-HelperThread-#2,5,main]. Will attempt to reconstruct, but this might mean that something bad is happening.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2020-04-16T03:33:34,344 WARN  (C3P0PooledConnectionPoolManager[identityToken->z8kflta8pplzokkop8au|76637162]-HelperThread-#1) [;;;] com.mchange.v2.async.ThreadPoolAsynchronousRunner - An Error forced the closing of Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta8pplzokkop8au|76637162]-HelperThread-#1,5,main]. Will attempt to reconstruct, but this might mean that something bad is happening.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2020-04-16T03:34:15,903 WARN  (Replication Thread) [;;;] com.vmware.horizon.datastore.IdmRMIAsynchronousCacheReplicator - 2 messages were discarded on replicate due to reclamation of SoftReferences by the VM. Consider increasing the maximum heap size and/or setting the starting heap size to a higher value.
2020-04-16T03:34:41,105 WARN  (AMQP Connection 127.0.0.1:5672) [;;;] com.vmware.horizon.messaging.provider.rabbitmq.RabbitMQMessagePublisher - publisher's [destination: -.analytics] connection shutdown by RabbitMQ. Reason: null
2020-04-16T03:34:04,787 ERROR (subscriber-thread-212) [;;;] com.vmware.horizon.messaging.provider.rabbitmq.MessagingUtils - Setting up messaging provider connection failed.
java.util.concurrent.TimeoutException: null

High memory usage alerts may be thrown by monitoring software such as Aria Operations.

Environment

VMware Aria / vRealize Automation 8.x
VMware Identity Manager 3.3.x

Cause

The following scenarios will cause issues with undersized VMware Identity Manager 3.3.x appliances:

• OAuth2RefreshToken issue as in KB 322682
• Large number of concurrent logins and number of users synced
• High frequency of domain sync
• Internal Postgres query cache
• Considerable number of SCIM APIs triggered by vRealize Automation
• DB Cache replication threads
• Configuration issues, such as not all IDM nodes added as connectors

Resolution

In the case of the OAuth2RefreshToken issue (/db disk filling up), please see the resolution in KB 322682

Please ensure the vIDM configuration is complete and accurate.

It is recommended to resize any vIDM 3.3.x appliance to the appropriate specification, at a minimum the below, to help prevent many other memory-related issues:

• 8 CPUs and 16 GB memory

 

Please see the Identity Manager docs for more sizing info.