High CPU utilization and Out of Memory messages in vIDM 3.3.x
search cancel

High CPU utilization and Out of Memory messages in vIDM 3.3.x

book

Article ID: 326077

calendar_today

Updated On:

Products

VMware VMware Aria Suite

Issue/Introduction

Symptoms:
High CPU utilization and OutOfMemory error seen in horizon.log , gradually the vIDM system may become unresponsive.

In the /opt/vmware/horizon/workspace/logs/horizon.log file, you see entries similar to:
2020-04-16T03:34:01,512 WARN  (C3P0PooledConnectionPoolManager[identityToken->z8kflta8pplzokkop8au|76637162]-HelperThread-#2) [;;;] com.mchange.v2.async.ThreadPoolAsynchronousRunner - An Error forced the closing of Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta8pplzokkop8au|76637162]-HelperThread-#2,5,main]. Will attempt to reconstruct, but this might mean that something bad is happening.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2020-04-16T03:33:34,344 WARN  (C3P0PooledConnectionPoolManager[identityToken->z8kflta8pplzokkop8au|76637162]-HelperThread-#1) [;;;] com.mchange.v2.async.ThreadPoolAsynchronousRunner - An Error forced the closing of Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta8pplzokkop8au|76637162]-HelperThread-#1,5,main]. Will attempt to reconstruct, but this might mean that something bad is happening.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2020-04-16T03:34:15,903 WARN  (Replication Thread) [;;;] com.vmware.horizon.datastore.IdmRMIAsynchronousCacheReplicator - 2 messages were discarded on replicate due to reclamation of SoftReferences by the VM. Consider increasing the maximum heap size and/or setting the starting heap size to a higher value.
2020-04-16T03:34:41,105 WARN  (AMQP Connection 127.0.0.1:5672) [;;;] com.vmware.horizon.messaging.provider.rabbitmq.RabbitMQMessagePublisher - publisher's [destination: -.analytics] connection shutdown by RabbitMQ. Reason: null
2020-04-16T03:34:04,787 ERROR (subscriber-thread-212) [;;;] com.vmware.horizon.messaging.provider.rabbitmq.MessagingUtils - Setting up messaging provider connection failed.
java.util.concurrent.TimeoutException: null


Environment

VMware vRealize Automation 8.x
VMware Identity Manager 3.3.x

Cause

The following scenarios will cause issues with undersized VMware Identity Manager 3.3.x appliances:

  • Internal Postgres query cache
  • Considerable number of SCIM APIs triggered by vRealize Automation
  • DB Cache replication threads
  • Number of concurrent logins and number of users synced

Resolution

It is recommended to resize any vIDM 3.3.x appliance to the appropriate specification, at a minimum the below, to help prevent this issue:

  • 8 CPUs ad 16 GB memory
Note: This is higher than the defaultĀ of 2 CPUs and 6 GB of memory, which is the current default hardware configuration in vIDM 3.3.1 and vIDM 3.3.2 when deployed through OVA.