Managing Kafka & Zookeeper: Locating KAFKA_HOME, checking status, topic administration, and message viewing

Products

• VMware Telco Cloud Service Assurance

Issue/Introduction

• Determine the KAFKA_HOME location
• How to check the status of Kafka and Zookeeper?
• Commands to list, delete and create topics in Kafka. 
• How to view messages of a topic in Kafka?

 

 

Environment

10.X

Resolution

Determine the KAFKA_HOME location:

Find the server.properties file, which is located in the KAFKA_HOME/config location:

• $ locate server.properties

How to check the status of Kafka and Zookeeper?

View the file server.properties

• $ less /opt/kafka/config/server.properties

  • listeners=SASL_PLAINTEXT://#.#.#.#:9092

  • zookeeper.connect=#.#.#.#:2181

Ports in use by kafka and zookeeper services can be identified within this file.There should be java processes listening at these ports.

1. Check the kafka port listed in the server.properties:

   $ sudo netstat -anp | grep 9092

2. The below output shows that there is a java process listening indicating good results.

   tcp6       0      0 :::9092                 :::*                    LISTEN      52922/java

3. Check to see if zookeeper process is listening to port 2181

   $ sudo netstat -anp | grep 2181

4. The below output shows that there is a java process listening indicating good results.

   tcp6       0      0 10.10.77.21:2181      :::*                    LISTEN      52926/java

5. Best practice on Linux OS is to systemctl to create services for Kafka and Zookeeper, and use these to start and stop the processes.

   See Set up services for Kafka and Zookeeper components using systemctl (Linux) so they will restart after system restart

6. Check the Kafka service status:

   $ sudo systemctl status kafka? kafka.service

7. Systemctl messages output to /var/log/messages:
   

   Dec 19 17:33:43 rpc11845 systemd: Stopping kafka.service...

   Dec 19 17:33:43 rpc11845 zookeeper-server-start.sh: [2019-12-19 17:33:43,586] INFO Processed session termination for sessionid: 0x100b5e538cf0003 (org.apache.zookeeper.server.PrepRequestProcessor)

   Dec 19 17:33:43 rpc11845 zookeeper-server-start.sh: [2019-12-19 17:33:43,589] INFO Closed socket connection for client /10.10.77.21:49764 which had sessionid 0x100b5e538cf0003 (org.apache.zookeeper.server.NIOServerCnxn)

   Dec 19 17:33:44 rpc11845 zookeeper-server-start.sh: [2019-12-19 17:33:44,179] INFO Processed session termination for sessionid: 0x100b5e538cf0002 (org.apache.zookeeper.server.PrepRequestProcessor)

   Dec 19 17:33:44 rpc11845 zookeeper-server-start.sh: [2019-12-19 17:33:44,181] INFO Closed socket connection for client /10.10.77.21:49662 which had sessionid 0x100b5e538cf0002 (org.apache.zookeeper.server.NIOServerCnxn)

   Dec 19 17:33:46 rpc11845 systemd: Started kafka.service.

 Commands to list, delete and create topics in Kafka:

• List Topics: (All on one line)
  $ export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf/opt/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181
  
  
• Delete topics (All in one line):
  $ export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf
/opt/kafka/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic discoveryTopic
/opt/kafka/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic monitoringTopic
  
  
• Create topics (All in one line):  
  $ export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf
/opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic monitoringTopic
/opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic discoveryTopic 

How to view messages of a topic in Kafka?

• View topic messages:
  $ export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_client_jaas.conf
$ cd /opt/kafka/bin/
$ ./kafka-console-consumer.sh --topic discoveryTopic -from-beginning --consumer.config=consumer.properties  --bootstrap-server=<server_IP/name>:9092
$ ./kafka-view-discoveryTopic.sh | head

 

Additional Information

Official Apache Kafka page
Official Zookeeper page