Kafka virtualization using DevTest

Products

• Service Virtualization

Issue/Introduction

Steps to perform virtualization of Kafka topics

Environment

DevTest 10.7.x

Resolution

DevTest – Kafka

Installing and starting Kafka on a Unix VM

1. Download Kafka from its official website

wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.13-3.0.0.tgz

wget https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz

2. Untar the downloaded Kafka file

tar -xvzf ~/Downloads/kafka_2.13-3.0.0.tgz

3. Rename the untarred kafka folder to kafka

mv kafka_2.13-3.0.0 kafka

4. Start Zookeeper

~/kafka/bin/zookeeper-server-start.sh ~/kafka/config/zookeeper.properties &

5. Start Kafka

~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties &

6. Create Topics

~/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic first_topic --create --partitions 3 --replication-factor 1

7. List Topics

~/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

8. Produce messages

~/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic

9. Consume messages

~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic

 

DevTest & Kafka Setup

1. Import the Kafka Project in your DevTest workstation.
2. Double click on the project config file and right click on Kafka Connection and then click on add to add the kafka connection
3. Add the Kafka connection details as below and connect.                             
4.    
5. Copy the following third-party jar files to the 

<LISA_HOME>/lib/shared directory:

• kafka-clients-2.5.0.jar
• lz4-java-1.4.jar
• snappy-java-1.1.7.1.jar

 

6. Create 4 topics -:

• Input
• Output
• Input
• Output

Two types of Kafka Recording can be done -:

 

• Proxy Recorder – allows you to do things without interfering with actual service.
• Topic Recorder – you will have to stop the live service because you are publishing and subscribing from the topics which live system uses.

Proxy Recorder

 

Click on VSE Recorder and put on the details as below. Make sure to select the Transport protocol as “Apache Kafka” and then click on next.

 

 

Select “Proxy Recorder” and give the topics names on the Proxy and live Request/ Response.

Click on next and we are ready to record the back-end service.

Now go to the workstation and run the Interactive Test Run (ITR) for “driver-proxy.tst”

 

Now check the Proxy recorder and you would see 5 request and 5 responses.  

Click next and add data protocol as XML Data Protocol and then click next and Finish.

We now successfully recorded the service which interacted with Kafka topics like Input and Output. Model file created named RecordProxy-04.vsm.

Deploy the RecordProxy-04.vms to the VSE

Check in the portal if the service got deployed.

Now run the ITR on driver-proxy.tst to check if the RecordProxy-04 service gets the traffic or not.

So now all the traffic of the test goes to the virtual service instead of live service.

Topic Recorder

Let the Topic Recorder selected and give the Consume Topic as Input and Send Topic as Output.

Service is ready to be recorded.

 

Run the ITR on “driver.tst”

 

Add the Data protocol as XML Data Protocol and click on next and finish.

 

Model file RecordTopic-04.vsm got created. Deploy it on the VSE.

Check for the service deployment on the Devtest portal. Note the transaction count in RecordTopic-04 is 0 and actual service i.e. kafka-demo-service is 20.

 

So in order to avoid confusion as now both live and virtual service is consuming from same topic , it is good idea to stop the actual kafka-demo-service.

Now run ITR on driver.tst and check the transaction count of the RecordTopic-04 virtual service.

You can see that now the transaction is going to the virtual service instead of live service