VMware Smart Assurance Smarts 10.x: How to set up services for Kafka and Zookeeper components using systemctl (Linux) so they will restart after system restart?

Products

• VMware Smart Assurance

Issue/Introduction

The purpose of this article is to provide a method to add Kafka and Zookeeper as services using systemctl on Redhat Linux compatible kernels 
In addition the steps here will automate the restart of kafka and zookeeper after a system restart.

Note: This procedure was tested with Smarts 10.1 on CentOS Linux release 7.7.1908 (Core)

Symptoms:

In Smarts 10.x Kafka and Zookeeper are considered 3^rd party products by VMware and not bundled with the install of Smarts Server Manager. However they are required if you enable VeloCloud SD-WAN discovery and monitoring. 

It has been noted that installing Kafka and Zookeeper do not install as services by default, so one would have to restart zookeeper and kafka manually after a server restart.

Environment

VMware Smart Assurance - SMARTS

Resolution

Below are the steps to follow to add services for Kafka and Zookeeper 

1.  Kafka is generally download and extracted from tgz file to a directory on Linux machine.
Example: /opt/kafka_2.11-2.0.0
 

2.  Add Unix group/user for Kafka and set directory ownership to this use with following commands:      groupadd kafka
useradd -g kafka kafka
passwd kafka
 

3.  Change directory ownership. mkdir will fail if directory already exists.
mkdir /var/zookeeper
chown -R kafka:kafka /var/zookeeper
mkdir /var/log/zookeeper
chown -R kafka:kafka /var/log/zookeeper
chown -R kafka:kafka /opt/kafka_2.11-2.0.0

 

4. Create file kafka.service under /etc/systemd/system, add the following lines & save the file:

[Unit]

Requires=zookeeper.service

After=zookeeper.service

[Service]

Environment="KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka_2.11-2.0.0/config/kafka_server_jaas.conf"

Type=simple

User=kafka

ExecStart=/bin/sh -c '/opt/kafka_2.11-2.0.0/bin/kafka-server-start.sh /opt/kafka_2.11-2.0.0/config/server.properties > /opt/kafka_2.11-2.0.0/kafka.log 2>&1'

ExecStop=/opt/kafka_2.11-2.0.0/bin/kafka-server-stop.sh

Restart=on-abnormal

[Install]

WantedBy=multi-user.target

 

5.  Create file zookeeper.service under /etc/systemd/system, add the following lines & save the file: 

[Unit]

Requires=network.target remote-fs.target

After=network.target remote-fs.target

[Service]

Environment="KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka_2.11-2.0.0/config/zookeeper_jaas.conf"

Type=simple

User=kafka

ExecStart=/opt/kafka_2.11-2.0.0/bin/zookeeper-server-start.sh /opt/kafka_2.11-2.0.0/config/zookeeper.properties

ExecStop=/opt/kafka_2.11-2.0.0/bin/zookeeper-server-stop.sh

Restart=on-abnormal

[Install]

WantedBy=multi-user.target

 

6.  Start and enable the services with following commands so they are automatically restarted when the system is restarted .    
systemctl start zookeeper.service
systemctl enable zookeeper.service
systemctl start kafka.service
systemctl enable kafka.service

 

7.  Check the status of Kafka & Zookeeper services with below commands: 

systemctl status kafka
 kafka.service
 Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: disabled)
 Active: active (running) since Thu 2019-12-12 16:19:26 GMT; 1 weeks 0 days ago
 Process: 57455 ExecStop=/opt/kafka/bin/kafka-server-stop.sh (code=exited, status=1/FAILURE)

 Main PID: 52923 (sh)
   CGroup: /system.slice/kafka.service

           +-52923 /bin/sh -c /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties ...

           +-52926 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccu...

 

 

systemctl status zookeeper
 zookeeper.service
 Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: disabled)
 Active: active (running) since Thu 2019-12-12 16:19:26 GMT; 1 weeks 0 days ago
 Process: 45496 ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh (code=exited, status=1/FAILURE)

 Main PID: 52922 (java)

   CGroup: /system.slice/zookeeper.service

           +-52922 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeap...

Below is the command to restart Kafka service with systemctl, it does not print a status message. 
 sudo systemctl restart kafka

Additional Information

For additional troubleshooting steps refer KB article 
https://kb.vmware.com/s/article/76635 - VMware Smart Assurance Smarts 10.x: How to check status and details of Kafka component?

Impact/Risks:

VMware does not warrant the steps used in this article, in case of any issues within the OS contact the respective OS vendor or issues with installing Kafka/ Zookeeper consult with apache.org. 

In case of problems, errors from systemctl are logged in /var/log/messages for reference.