Get error "failed to stop container MOI2WT_MM2", when attempting to stop and restart MOI

Products

• Mainframe Operational Intelligence

Issue/Introduction

When attempt to stop and restart MOI.  There is some errors on the stopping process as follows: 

[INFO] DoShutdown/453: Stopping database containers
Stopping cassandra_cassandra3_1 ... done
Stopping cassandra_cassandra2_1 ... done
Stopping cassandra_cassandra1_1 ... done
Stopping confluent_confluent_1 ... done
Stopping kafka_kafka_1 ... done
Removing cassandra_cassandra3_1 ... done
Removing cassandra_cassandra2_1 ... done
Removing cassandra_cassandra1_1 ... done
Network ca_itoa_network is external, skipping
Removing confluent_confluent_1 ... done
Network ca_itoa_network is external, skipping
Removing kafka_kafka_1 ... done
Network ca_itoa_network is external, skipping
Remove dangling containers
WARN[0010] StopSignal SIGTERM failed to stop container MOI2WT_MM2 in 10 seconds, resorting to SIGKILL
Error: cannot remove container as it could not be stopped: container exited prematurely, exit code could not be retrieved: internal libpod error
Remove dangling containers failed, retry later...
89932a7d4388
Complete moi-shutdown

Environment

MOI 2.1 SP9 or later

Resolution

We can see in the error messaging that it's referencing the MOI2WT_MM2 process during the MOI shutdown.

Please perform the following steps:

1.  Check if the MOI2WT_MM2 container is still running/exited by running the command:
docker ps -a | grep MOI2WT_MM2

2.  If the above command shows that the process is still there and in exited state, then clean it up using the following commands:
docker stop MOI2WT_MM2 && docker rm MOI2WT_MM2

3. Run the command to ensure that the above container is not seen:
docker ps -a    

4.  If you still see the container, just repeat the step: 
docker stop MOI2WT_MM2 && docker rm MOI2WT_MM2

5.  Next ensure that MOI is up and running.

6.  Finally, start the MOI mirrormaker process using the following command:
systemctl start moi2wt-mm2

This should start MOI mirrormaker correctly. 

Note that After a period of time, the status of the this container will be unhealthy, but that should be OK.

Our MirrorMaker (moi2wt-mm2) container service configuration is intended only for data mirroring from MOI to WatchTower (WT).  While the container is successfully mirroring data in this direction (MOI to WT), it is also attempting to establish a connection from WT to MOI.  Because this reverse path is not supported in our setup, it is generating exceptions in the logs, which is causing the moi2wt-mm2 container to report an unhealthy state.

It should be noted that these exceptions do not impact our primary data flow.  Since WT to MOI mirroring is not an intended purpose of this script, these logs and the resulting unhealthy state for moi2wt-mm2 can be disregarded for the time being.