Dollar Universe freezes roughly every 5 hours, needs to be restarted.

Products

• CA Automic Dollar Universe

Issue/Introduction

$Universe freezes in ~5 hour intervals with the following symptoms

• high load on uxioserv process
• high number of opened files ("lsof -u universe | wc -l" is greater 11k)
• $U can't be stopped via unistop, needs to be forced
• product log states
  | 2021-05-30 23:10:58 |WARN |X|IO |pid=18803.140327344121600| check_art_lock            | u_fmcx60[6542] is free (no file lock)
  | 2021-05-30 23:10:58 |INFO |X|IO |pid=18803.140333056595712| dispatch thp              | Thread pool max reached. Connection in queue.
• Even when force stopped, there are multiple processes left like

Environment

Release : 6.x

Component : DOLLAR UNIVERSE

Cause

In review of logs we found that

• uxioserv CPU usage was very high
• uxlst command been frequently executed

Log snippets

• Log name: top_*.txt

PID      MEM     CPU    CMD
18803 18.161g 105.56 uxioserv
18803 18.161g 120.72 uxioserv
18803 18.161g 120.92 uxioserv

• Log name: pstree_*.txt

~Total execution count =507

  |   |-sshd,21141
  |   |   `-sshd,21146
  |   |       `-uxlst,21147 sw universe pi4.migros.ch MIGROS bin uxlst ctl EXP UPR=* MU=* STATUS=E,A FULL

The issue originated because of excessive un-monitored call of uxlst remotely i.e. uxlst ctl upr=* ses=* mu=* status=E,A which haven't finished. This command provides ALL job executions in Running or Aborted, that is not advised, as in busy environment there will be large number of jobs in prior mentioned state, and command will take more time to complete.

Resolution

We will recommend below suggestion to overcome the issue

• There is native supervision in Dollar Universe which would be best used for this matter. (preferred)
• The supervision script/remote system need to enhance to consider not firing new request until previous is finished execution.
• Increase the time gap between successive execution of uxlst command.