DQM became irresponsive and Dollar Universe stopped submitting Jobs.
Error in universe.log :
| 2020-09-09 13:30:02 |ERROR|X|DQM|pid=16587.370270| u_dqm_cli_thread_trt | Memory error (32774)
| 2020-09-09 13:30:14 |ERROR|X|DQM|pid=16587.370272| u_dqm_cli_thread_trt | Memory error (32774)
| 2020-09-09 13:30:14 |ERROR|X|IO |pid=16459.55| owls_connect_auth | k_connect_auth_timeout(BEX0454Z5A/DQM) returns error [-1]
| 2020-09-09 13:30:14 |ERROR|X|IO |pid=16459.55| o_callsrv_connect_r | Connection error 0 [Hello request error]
Unknown memory issue
Release : 6.10
Component : DOLLAR UNIVERSE
Workaround: restart the node when this issue occurs.
RCA unknown as no traces are available, we suspect that DQM hit a system / process memory limitation but we were unable to found out which was the limitation.