JGroups fails any Clarity service with "thread pool is full" errors

Products

• Clarity PPM On Premise
• Clarity PPM SaaS

Issue/Introduction

While starting all the clarity services at once below error is seen 

FATAL YYY-MM-DD SS:SS:12,029 [ucast-receiver-7,xxxxxx] protocols.UDP (none:none:none:none)xxxxxxxx: thread pool is full (max=8, active=8); thread dump (dumped once, until thread_dump is reset)

Steps to Reproduce:

1. Set up a clarity server with below configuration
   • Total Memory around 32 GB on windows OS
   • Configure two app service with about 6.5 GB JVM
   • Configure one bg service with about  6.5 GB JVM
   • Configure CSA and beacon with default JVM 
2. Start all the services by executing the command 
   • Service start all 

Expected Results: All the clarity services should start without any error on the app ca or bg ca log

Actual Results: Below Fatal logs are seeing in the app ca and bg ca logs  

FATAL 2023-04-01 16:26:12,029 [ucast-receiver-7,xxxxxx] protocols.UDP (none:none:none:none)xxxxxxxx: thread pool is full (max=8, active=8); thread dump (dumped once, until thread_dump is reset)

Environment

Release: 16.x

Cause

DE69252 - The thread pool error comes up when there is thread crunch while starting all the services at once. 

Resolution

DE69252 overall improvement has been done the way caching is managed in clarity. This may be happening past 16.1.3 on systems with lack of CPU resources.

Workaround:

• Ensure to have adequate memory for OS
• Ensure sufficient CPU available in the system and refer to Architecture and Sizing Recommendations
• Ensure Java parameters in the JVM is rightly configured
• Try starting one service at a time and observe if the thread pool error is observed