ZMSSTART no longer sending data to MOI

Products

• Mainframe Operational Intelligence

Issue/Introduction

Situation where one LPAR in a multi-LPAR plex was not sending data to MOI.   So, cycled ZMSSTART and it appeared to be healthy.   However, still not receiving data from the offending LPAR.  SYSVIEW CADCONS command shows ZMSSTART ACTIVE state, but no bytes being transmitted.   Also attempted to cycle of SYSVIEW but same issue persisted. 

 

Environment

Release : 2.1

Resolution

Background information:

If the Message Server is losing the connection to the MOI system it continues to collect the metric values and "buffers" them in the USS file. But if the connection is not reinstated in a certain period of time this file will get full and the Message Server stops the data collection.  Unfortunately the loss of the connection is not reported in the job log of the Message Server - but there will be messages when the storage becomes full.   

This buffering functionality is very important because it allows restarting MOI without losing data. but it may happen that this resuming of the data will not work after the file gets full. So often the only way to recover from this is to delete this KahaDB directory. But it may also be that this interrupt has caused a hang situation on the MOI side.

So could you try these steps to resolve:

-   Stop ZMSSTART on the Lpar with the problem
-   Restart the Message Service Hub in the MOI Appliance Controller
-   Start ZMSSTART on the z/OS system

-   If this does not resolve the situation stop ZMSSTART again
-   Do a complete MOI restart - the data from the other 3 Lpars will not be lost as the buffered data will be processed when the connection is reestablished.
-   Start ZMSSTART again and check if the data starts to flow

-   If the does not work stop ZMSSTART again
-   Delete the KahaDB directory in the USS directory
-   Restart the ZMSSTART

Note:  To better understand and analyze what has caused this situation.  It would be great if you could create a MOI dump and upload it to the together with the complete set of job logs of the ZMSSTART and the ZDMVPROC Started Tasks, plus the server.log files from the '../cfgb/instance/messageService/logs' directory.

Customer noted the following steps to resolve:

Shutdown ZMSSTART on SYSF
Deleted all the files in  SYSF /apps/ccs/ccsuss/cfgb/instance/messageService/brokerData/embeddedMaasBroker…/KahaDB
Started ZMSSTART - It started up immediately 

SYSF SYSVIEW command CADCONS – now looks better and xmit numbers are increasing.   Note that it took several minutes for the data to show up on the MOI webpage, even though the xmit numbers were increasing fairly quickly.

Additional question....To restart the Message Service Hub in the MOI Appliance Controller.  Is this only ONE of the "micro services"  OR  is this the same as clicking the green button on MOI Appliance Controller where ALL the micro services are restarted?    If this is done, either with ONE micro service or the whole lot of them.

Yes - The green button on the left top corner of the Appliance Controller Dashboard display stops and start all microservices. It is also possible to restart single microservices in the right panel titled Microservices. There is red button that allows to stop selected microservices.  After the service has exited a green button will allow to restart it.