Managing Kafka & Zookeeper: Locating KAFKA_HOME, checking status, topic administration, and message viewing

Products

• VMware Smart Assurance
• Network Observability

Issue/Introduction

This article covers various topics and initial configuration details for Kafka and zookeeper. 

• Determine the KAFKA_HOME location
• How to check the status of Kafka and Zookeeper?
• Commands to list, delete and create topics in Kafka. 
• How to view messages of a topic in Kafka?

Environment

All Supported Smarts releases

Resolution

Determine the KAFKA_HOME location:

Find the server.properties file, which is located in the KAFKA_HOME/config location:

• $ locate server.properties

How to check the status of Kafka and Zookeeper?

View the file server.properties

• $ less /opt/kafka/config/server.properties

  • listeners=SASL_PLAINTEXT://#.#.#.#:<PORT Like 9092>

  • zookeeper.connect=#.#.#.#:<PORT Like 2181>

Ports in use by kafka and zookeeper services can be identified within this file .There should be java processes listening at these ports.

1. Check the kafka port listed in the server.properties:

   $ sudo netstat -anp | grep <PORT Like 9092>

2. The below output shows that there is a java process listening indicating good results.

   tcp6       0      0 :::<PORT Like 9092>                 :::*                    LISTEN      <PID Like 52922>/java

3. Check to see if zookeeper process is listening to port 2181

   $ sudo netstat -anp | grep <PORT Like 2181>

4. The below output shows that there is a java process listening indicating good results.

   tcp6       0      0 #.#.#.#:<PORT Like 2181>      :::*                    LISTEN      <PID Like 52926>/java

5. Best practice on Linux OS is to systemctl to create services for Kafka and Zookeeper, and use these to start and stop the processes.

   See Set up services for Kafka and Zookeeper components using systemctl (Linux) so they will restart after system restart

6. Check the Kafka service status:

   $ sudo systemctl status kafka.service

7. Systemctl messages output to /var/log/messages:
   

   <Date & Time> <Host> systemd: Stopping kafka.service...
   <Date & Time> <Host> zookeeper-server-start.sh: [<Date & Time>,586] INFO Processed session termination for sessionid: #SESSIONID#(org.apache.zookeeper.server.PrepRequestProcessor)
   <Date & Time> <Host> zookeeper-server-start.sh: [<Date & Time>,589] INFO Closed socket connection for client /#.#.#.#:<PORT> which had sessionid #SESSIONID# (org.apache.zookeeper.server.NIOServerCnxn)
   <Date & Time> <Host> zookeeper-server-start.sh: [<Date & Time>,179] INFO Processed session termination for sessionid: #SESSIONID# (org.apache.zookeeper.server.PrepRequestProcessor)
   <Date & Time> <Host> zookeeper-server-start.sh: [<Date & Time>,181] INFO Closed socket connection for client /#.#.#.#:<PORT> which had sessionid #SESSIONID# (org.apache.zookeeper.server.NIOServerCnxn)
   <Date & Time> <Host> systemd: Started kafka.service.

 Commands to list, delete and create topics in Kafka:

• List Topics: (All on one line)
  

  $ export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf/opt/kafka/bin/kafka-topics.sh --list --zookeeper #.#.#.#:2181

• Delete topics (All in one line):
  

  • $ export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf

  • /opt/kafka/bin/kafka-topics.sh --delete --zookeeper #.#.#.#:2181 --topic discoveryTopic

  • /opt/kafka/bin/kafka-topics.sh --delete --zookeeper #.#.#.#:2181 --topic monitoringTopic

• Create topics (All in one line):  
  

  • $ export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf

  • /opt/kafka/bin/kafka-topics.sh --create --zookeeper #.#.#.#:2181 --replication-factor 1 --partitions 1 --topic monitoringTopic

  • /opt/kafka/bin/kafka-topics.sh --create --zookeeper #.#.#.#:2181 --replication-factor 1 --partitions 1 --topic discoveryTopic 

How to view messages of a topic in Kafka?

• View topic messages:
  

  • $ export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_client_jaas.conf

  • $ cd /opt/kafka/bin/

  • $ ./kafka-console-consumer.sh --topic discoveryTopic -from-beginning --consumer.config=consumer.properties  --bootstrap-server=<server_IP/name>:9092

  • $ ./kafka-view-discoveryTopic.sh | head

 

Additional Information

Official Apache Kafka page
Official Zookeeper page