Node hangs with Thread pool max reached generated when launching uxshw cal or uxtrace

Products

• Automic Dollar Universe

Issue/Introduction

On a Dollar Universe node, the following errors and warnings appear in the universe.log:

| INFO |X|IO | dispatch thp | Thread pool max reached. Connection in queue
| WARN |X|IO | u_io_thread_trt | Closing client connected for 61 minutes without activity

This occurs when the command "uxshw cal mu=*" is launched against a node with many calendars and years (file u_fmca60.dta size of 8MB containing 270 calendars, most of them with around 35 years each).

As a result, the node hangs totally during the time the command is running, ie. when a uxtrace is launched against the node.

Environment

Dollar Universe 6.x or 7.x with many calendars / years defined.

Cause

Thread Pool Exhaustion (DE205250): Large queries (e.g., uxshw cal mu=*) against massive calendar files (u_fmca60.dta) saturate the IO server thread pool.

Resolution

Workaround: 

If an upgrade is not immediately possible, follow these steps:

1. Increase the node configuration parameter U_IO_THREADPOOL_MAX to 500 (or 1024 for high-activity nodes), this can be done in the node settings - technical settings via UVC
2. Clean up historical or unnecessary years in calendars to reduce the size of the u_fmca60.dta file.
3. Use specific Management Unit (MU) filters instead of wildcards (mu=*) in command-line "uxshw cal" commands.

Resolution: 

This problem is currently being worked on by Engineering and the planned fix delivery method will be communicated ASAP

Additional Information

We suggest you to subscribe to this article for future updates with fix availability.