DUAS: Node connected with warning "Thread pool max reached"

Products

• CA Automic Dollar Universe

Issue/Introduction

Node suddenly goes to "Connected with Warning" status.

Running the following commands, shows that the IO engine for one area has stopped:
- uxlst atm all
- uxlst fnc fnc=* exp
- uxlst fnc fnc=* sim
- uxlst fnc fnc=* app
- uxlst fnc fnc=* int


In universe.log at the time the node changes status many "Thread pool max reached" messages for the I-Area:
| 2020-03-30 00:01:33 |INFO |I|IO |pid=10488.139663793493824| dispatch thp   | Thread pool max reached. Connection in queue.
| 2020-03-30 00:02:03 |INFO |I|IO |pid=10488.139663793493824| dispatch thp   | Thread pool max reached. Connection in queue.


"netstat -na" shows the ephemeral ports all in CLOSE_WAIT status between the local IO port of the area that has stopped and another node's IP address.
tcp       48      0 10.200.100.1:10602     10.202.101.2:44024     CLOSE_WAIT 
tcp       48      0 10.200.100.1:10602     10.202.101.2:33202     CLOSE_WAIT 
tcp       48      0 10.200.100.1:10602     10.202.101.2:46692     CLOSE_WAIT

 

Environment

Release : 6.x
Component : DOLLAR UNIVERSE

Cause

Firewall is blocking network traffic between two DUAS nodes or a DUAS node and UVMS, resulting in an exhaustion of all the ephemeral ports, as seen by them being in a CLOSE_WAIT status, and as seen with the "Thread pool max reached. Connection in queue." messages in the universe.log file.

Resolution

Ensure that network traffic is open between all DUAS nodes and between DUAS nodes and UVMS. In both directions and for all ports being used!