Node suddenly goes to "Connected with Warning" status.
Running the following commands, shows that the IO engine for one area has stopped:
- uxlst atm all
- uxlst fnc fnc=* exp
- uxlst fnc fnc=* sim
- uxlst fnc fnc=* app
- uxlst fnc fnc=* int
In universe.log at the time the node changes status many "Thread pool max reached" messages for the I-Area:
| 2020-03-30 00:01:33 |INFO |I|IO |pid=10488.139663793493824| dispatch thp | Thread pool max reached. Connection in queue.
| 2020-03-30 00:02:03 |INFO |I|IO |pid=10488.139663793493824| dispatch thp | Thread pool max reached. Connection in queue.
"netstat -na" shows the ephemeral ports all in CLOSE_WAIT status between the local IO port of the area that has stopped and another node's IP address.
tcp 48 0 10.200.100.1:10602 10.202.101.2:44024 CLOSE_WAIT
tcp 48 0 10.200.100.1:10602 10.202.101.2:33202 CLOSE_WAIT
tcp 48 0 10.200.100.1:10602 10.202.101.2:46692 CLOSE_WAIT
Release : 6.x
Component : DOLLAR UNIVERSE