DUAS: Node connected with warning "Thread pool max reached"

Products

• CA Automic Dollar Universe

Issue/Introduction

Node suddenly goes to "Connected with Warning" status.

Running the following commands, shows that the IO engine for one area has stopped:
- uxlst atm all
- uxlst fnc fnc=* exp
- uxlst fnc fnc=* sim
- uxlst fnc fnc=* app
- uxlst fnc fnc=* int


In universe.log at the time the node changes status many "Thread pool max reached" messages for the I-Area:
| 2020-03-30 00:01:33 |INFO |I|IO |pid=10488.139663793493824| dispatch thp   | Thread pool max reached. Connection in queue.
| 2020-03-30 00:02:03 |INFO |I|IO |pid=10488.139663793493824| dispatch thp   | Thread pool max reached. Connection in queue.


"netstat -na" shows the ephemeral ports all in CLOSE_WAIT status between the local IO port of the area that has stopped and another node's IP address.
tcp       48      0 10.200.100.1:10602     10.202.101.2:44024     CLOSE_WAIT 
tcp       48      0 10.200.100.1:10602     10.202.101.2:33202     CLOSE_WAIT 
tcp       48      0 10.200.100.1:10602     10.202.101.2:46692     CLOSE_WAIT

 

Environment

Release : 6.x
Component : DOLLAR UNIVERSE

Cause

Firewall was blocking network traffic between two nodes, resulting in an exhaustion of all the ephemeral ports, as seen by them being in a CLOSE_WAIT status, and as seen with the "Thread pool max reached. Connection in queue." messages in the universe.log file.

Resolution

Ensure that network traffic is open between the two nodes.