MOI to WatchTower Data Streaming Mirroring Solution

Products

• WatchTower
• WatchTower Platform
• Mainframe Operational Intelligence

Issue/Introduction

This guide describes the steps to externalize Kafka from WatchTower (WT) using a service script and set up MirrorMaker 2 (MM2) on MOI to mirror data securely.

Environment

MOI 2.1 SP8 or later

WT 1.3 or later 

Cause

 

Resolution

Prerequisites

For WatchTower (WT):

WT 1.3 must be installed and running with HUB in SSL mode

• Ensure datastream-tls-secret is deployed in the WT namespace

For MOI: 

• MOI SP8 is up and running with HUB and MSS connected in SSL

 

Enable Data Streaming MOI to WatchTower

Section 1:  Configuring WatchTower to receive data from MOI

1.  Download and extract the WT-kafka-external-service.tgz archive on the host with access to WT namespace:

tar -zxvf WT-kafka-external-service.tgz
cd kafka-external-service
chmod +x create-kafka-external-service.sh

2.  Run the script in dry-run mode to validate the YAML files before applying:

./create-kafka-external-service.sh \
  --namespace <WT_NAMESPACE> \
  --service-type <LoadBalancer|NodePort> \
  --external-port <EXTERNAL_PORT> \
  --external-host <EXTERNAL_HOST> \  --dry-run

Where:

• WT_NAMESPACE = Namespace where WT is installed

• SERVICE_TYPE = LoadBalancer or NodePort

• EXTERNAL_PORT = Port to expose Kafka

• EXTERNAL_HOST = DNS name pointing to external Kafka

Example:

./create-kafka-external-service.sh \
  --namespace wt-namespace \
  --service-type loadbalancer \
  --external-port 61615 \
  --external-host kafka-external.wt.company.com \
  --dry-run

Where:

kafka-external.wt.company.com is the external-host FQDN example

Output:

Two YAMLs are generated:

• kafka-svc.yaml: Service definition for external Kafka

• kafka-sts-patch.yaml: Patch for Kafka StatefulSet to advertise external listener

Review the YAMLs before proceeding.

3.  Apply changes

Once yaml files are validated, rerun the script without --dry-run:

Depending on the --service-type, execute either Case 1 or Case2

Case 1: loadbalancer as service type

./create-kafka-external-service.sh \
  --namespace wt-namespace \
  --service-type loadbalancer \
  --external-port 61615 \
  --external-host kafka-external.wt.company.com

You will see a confirmation like:

[SUCCESS] Kafka external access configured successfully.

 

Case 2: nodeport as service type

./create-kafka-external-service.sh \
  --namespace wt-namespace \
  --service-type nodeport \
  --external-port 31315 \
  --external-host kafka-external.wt.company.com

You will see a confirmation like:

[SUCCESS] Kafka external access configured successfully.

Important (Post-Deployment - MUST DO)

• Run the following to get the

  • LoadBalancer IP (for Case 1):

    kubectl get svc kafka-external -n wt-namespace -o jsonpath={.status.loadBalancer.ingress[0].ip}

  • NodePort IP (for Case 2):

kubectl get nodes -n wt-namespace -o jsonpath="{.items[0].status.addresses[0].address}"

• Create a DNS entry mapping the LoadBalancer IP or the Node IP (depending on the service type chosen) to kafka-external.wt.company.com

One way to set it up is to add an /etc/hosts entry on the box where MOI is setup like below

xxx.xx.xx.xxx kafka-external.wt.company.com

 

Section 2:  Configure the MOI System needed to feed WatchTower

1. Download and extract the MOI-kafka-mirrormaker.tgz archive

cd /opt/

tar -zxvf MOI-kafka-mirrormaker.tgz
cd MM2
chmod +x MM2.sh

2. Create the HUB_STORE Directory

'mkdir /opt/MM2/HUB_STORE'

3. Copy TLS Secret Files from WatchTower

Copy the following files from the `datastream-tls-secret` in WatchTower:

 `broker.ks`
 `broker.ts`
 `password.properties`

To the directory: 
'/opt/MM2/HUB_STORE'  

4. Run the MM2 setup script

./MM2.sh \
  --external-port 61615 \
  --external-host kafka-external.wt.company.com

Important

The --external-port and --external-host used in the MM2.sh script must exactly match the values provided when running create-kafka-external-service.sh on the WatchTower side.

This ensures that MM2 can correctly connect to the exposed external Kafka endpoint and mirror the data.

This script:

• • Creates MOI2WT_MM2.properties
  • Sets up and enables a systemd service named moi2wt-mm2

Sample Output:

[SUCCESS] Created the moi2wt-mm2.service file successfully
[INFO] Enabling and starting moi2wt-mm2.service
[INFO] Service moi2wt-mm2 is active (running)

...

systemd[1]: Starting MirrorMaker 2 Service (MOI2WT_MM2)...
systemd[1]: Started MirrorMaker 2 Service (MOI2WT_MM2).

 

Section 3:  Confirm the data flow from MOI to WatchTower is working 

On MOI:

• Run the following command to confirm data is being mirrored to WatchTower:

docker logs MOI2WT_MM2 | grep "Committing offsets" | grep "MirrorSourceConnector"

• You should see logs similar to the following coming in every minute:

INFO [MirrorSourceConnector|task-0|offsets] WorkerSourceTask{id=MirrorSourceConnector-0} Committing offsets for 60 acknowledged messages

On WatchTower:

• Go to Dashboards in UI

• Confirm zos dashboards show incoming data

 

Section 4:  Move the MSS feed from MOI to WatchTower

After 2 weeks, confirm the following prior to running decommissioning steps:

• Check if the system is running healthy and WT received at least 2 weeks of data.
• Confirm if anything else to be validated by Broadcom Team

Once confirmed, perform these steps:

• Switch from MSS to WT HUB.   This information is available within the WatchTower documentation within the "Complete MS Server Configuration" section. 

• On WT:

  • Delete the Kafka external service:

kubectl delete svc kafka-external -n <WT_NAMESPACE>

• On MOI:
  • Stop MirrorMaker on MOI:

docker stop MOI2WT_MM2 && docker rm MOI2WT_MM2

systemctl stop moi2wt-mm2

systemctl disable moi2wt-mm2

rm /usr/lib/systemd/system/moi2wt-mm2.service