The MOI DSP docker container uses an SSH Daemon to talk to the world outside the DSP docker container and run utility processes that are needed for machine learning.
If the SSHD fails for some reason, these utility processes will not run.
As an example of the impact of the SSHD The Mbuild process that builds a new machine learning model is scheduled to run every day. If the SSHD dies for some unknown reason, Mbuild will not run. If Mbuild does not run for a lengthy period of time, the Green Highways in MOI will revert back to the EWMA learning mode and no alerting will occur in MOI. The SSHD has to be restarted and the Mbuild will have to run for a period of a week or more to switch back to KDE mode and resume alerting.
The steps to restart the SSHD are contained in the Resolution of this Knowledge Doc.
Mainframe Operational Intelligence 2.0
How to restart the SSH Daemon to ensure that the dsp machine learning microservice utility processes run as scheduled:
Check the status of the dsp docker container:
docker ps -a|grep dsp
c64b93c519a1 mtcoi.packages.ca.com/x86_64/dsp:02.0.06-202111161001 "/bin/sh -c '[[ -f..." 3 days ago Up 3 days (healthy)
Stop the dsp docker container:
bin/plex down -- dsp
Start the dsp docker container:
bin/plex up -d -- dsp
Ensure that the dsp microservice becomes healthy after 5 to 10 mins:
docker ps -a | grep dsp
If the Mbuild process is the one that didn't run due to the SSHD dying, the next day or day and a half after stopping and starting the dsp docker container, run these commands that verify that the machine learning model build (MBuild) is running successfully on a scheduled basis which indicates that the SSH Daemon is alive:
To run a quick check on the status of the MBuild process in MOI, run the following two commands:
ls -la /var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log*
zgrep "Successfully" /var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log*
MOI Support may request that the output of these two commands be uploaded to an open MOI Support Case.
Example command output from your MOI appliance that your commands should produce:
[[email protected] dsp_logs]# ls -la /var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log*
-rw-r--r--. 1 ca_itoa ca_analytics 1332 Dec 9 10:19 /var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211209-091553.gz
-rw-r--r--. 1 ca_itoa ca_analytics 1336 Dec 9 23:08 /var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211209-215029.gz
-rw-r--r--. 1 ca_itoa ca_analytics 1335 Dec 10 23:00 /var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211210-215031.gz
-rw-r--r--. 1 ca_itoa ca_analytics 1334 Dec 11 22:46 /var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211211-215029.gz
-rw-r--r--. 1 ca_itoa ca_analytics 1333 Dec 12 22:55 /var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211212-215035.gz
[[email protected] dsp_logs]# zgrep "Successfully finished NightlyMBuild" /var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log*
/var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211209-091553.gz:ADMIN: 2021-12-09-10:19:56.829 P01866/Thr00 Successfully finished NightlyMBuild
/var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211209-215029.gz:ADMIN: 2021-12-09-23:08:20.854 P09364/Thr00 Successfully finished NightlyMBuild
/var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211210-215031.gz:ADMIN: 2021-12-10-23:00:07.501 P05615/Thr00 Successfully finished NightlyMBuild
/var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211211-215029.gz:ADMIN: 2021-12-11-22:46:34.009 P01449/Thr00 Successfully finished NightlyMBuild
/var/opt/moi/data2/profiler/var/opt/CA/itoa/dsp/dsp_logs/mbuild.log.20211212-215035.gz:ADMIN: 2021-12-12-22:55:46.243 P25315/Thr00 Successfully finished NightlyMBuild