HQ154004: Failed to deliver message, Failed to acquire a permit within 5 MINUTES

Products

• CA Identity Manager
• CA Identity Governance
• CA Identity Portal
• CA Identity Suite

Issue/Introduction

Task performance in the CA Identity Manager vApp (Virtual Appliance) has deteriorated and CPU usage is high.  The following errors appear in the logs:

2021-01-07 09:35:57,949 ERROR [org.hornetq.ra] (Thread-56 (HornetQ-client-global-threads-878909203)) HQ154004: Failed to deliver message: javax.ejb.EJBException: JBAS014516: Failed to acquire a permit within 5 MINUTES

 

Environment

Release : 14.3 CP2

Component : CA IDENTITY SUITE (VIRTUAL APPLIANCE)

Cause

Performance Issue

Resolution

The error below - illustrates a message bean instance depletion in Jboss\Hornetq.  If an instance is not available, HornetQ will try to wait for 5 minutes to see whether some instances return to the pool before the thread is failed.

2021-01-07 09:35:57,949 ERROR [org.hornetq.ra] (Thread-56 (HornetQ-client-global-threads-878909203)) HQ154004: Failed to deliver message: javax.ejb.EJBException: JBAS014516: Failed to acquire a permit within 5 MINUTES

 

Configuring MDB Maximum Sessions may help relieve this issue by creating more MDB Sessions.  The instructions to make the configuration changes are as documented in the product documentation (link below).

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-suite/14-3/virtual-appliance/administering-virtual-appliance.html#concept.dita_484b93c7f06198e8b27adcc2537229358eb17777_ConfigureMDBMaximumSessions

 

Extract: - In the Virtual Appliance, navigate to /opt/CA/VirtualAppliance/custom/IdentityManager.

1. Open jboss-ejb3.properties for editing.
2. Change the value of the maxSession property to the desired value.

maxSession=80

4. Restart Identity Manager.
restart_im

 

You will also need to increase the Max Pool Size.  This needs to be done using the jboss-cli (see details below).  

1. First you will need to create a user in the ManagementRealm.  On the vApp Console logon as config.

Execute the following.

/opt/CA/wildfly-idm/bin > sudo /opt/CA/wildfly-idm/bin/add-user.sh

What type of user do you wish to add?
 a) Management User (mgmt-users.properties)
 b) Application User (application-users.properties)
(a): a

Enter the details of the new user to add.
Using realm 'ManagementRealm' as discovered from the existing property files.
Username : mark
Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
 - The password should not be one of the following restricted values {root, admin, administrator}
 - The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
 - The password should be different from the username
Password :
JBAS015266: Password must have at least 1 digit.
Are you sure you want to use the password entered yes/no? yes
Re-enter Password :
What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[  ]:
About to add user 'mark' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'mark' to file '/opt/CA/wildfly-idm/standalone/configuration/mgmt-users.properties'
Added user 'mark' to file '/opt/CA/wildfly-idm/domain/configuration/mgmt-users.properties'
Added user 'mark' with groups  to file '/opt/CA/wildfly-idm/standalone/configuration/mgmt-groups.properties'
Added user 'mark' with groups  to file '/opt/CA/wildfly-idm/domain/configuration/mgmt-groups.properties'
Is this new user going to be used for one AS process to connect to another AS process?
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
yes/no? no

2. Now login to the jboss-cli to change the standalone xml file’s setting for MDB pool size.
To start CLI 

cd /opt/CA/wildfly-idm/bin
./jboss-cli.sh
connect 
<enter the user you created above>

3. Once connected run this command:

/subsystem=ejb3/strict-max-bean-instance-pool=mdb-strict-max-pool:write-attribute(name=max-pool-size,value=150)

4. Type quit to exit the jboss cli.

Additional Information

Values stated above are based upon medium to large enterprise deployments.  Further optimization "tweaks" may be required for specific environments.