DevTest Kafka Avro Schema De-Serializer SSL configuration

Products

• Service Virtualization

Issue/Introduction

How to configure the 'Kafka Send Receive' step for SSL and the Confluent Avro Schema Registry over HTTPS. 

Environment

All supported DevTest releases and platforms.

Cause

N/A

Resolution

For access to the Schema Registry the Consumer in the Kafka step needs to be configured with the Confluent Kafka Avro Serializer and Deserializer:

Class Name: io.confluent.kafka.serializers.KafkaAvroDeserializer

The DevTest lib folder should have the external jars to implement this functionality. 

For example if the server is running Confluent 7.0.1, the following jars should be copied from the Confluent Kafka server installation: 

12/04/2021  12:19 PM           589,142 avro-1.10.1.jar
12/04/2021  10:14 AM        16,203,680 confluent-metrics-7.0.1-ce.jar
12/04/2021  12:19 PM            75,703 jackson-annotations-2.12.3.jar
12/04/2021  12:19 PM           365,220 jackson-core-2.12.3.jar
12/04/2021  12:19 PM         1,515,756 jackson-databind-2.12.3.jar
02/08/2022  02:55 PM           809,924 jdk8-2.5.2.jar
12/04/2021  12:19 PM            30,974 kafka-avro-serializer-7.0.1.jar
12/04/2021  10:14 AM         5,775,017 kafka-clients-7.0.1-ce.jar
12/04/2021  12:19 PM            23,334 kafka-json-schema-serializer-7.0.1.jar
12/04/2021  12:19 PM            10,108 kafka-json-serializer-7.0.1.jar
12/04/2021  12:19 PM           177,393 kafka-schema-registry-client-7.0.1.jar
12/04/2021  12:19 PM            32,248 kafka-schema-serializer-7.0.1.jar

DevTest Kafka step configuration for access to the topic over SSL is configured with the SSL context:

See:  Section "Apache Kafka Assets" in the documentation of the DevTest release you are running.

 

 

DevTest Kafka step configuration for access to the Schema Registry over HTTPS is configured in the Consumer:

 

 

As the class and jars being used for the Deserializer are from Confluent, follow the Confluent documentation for the properties that can be used.

In this case: 
schema.registry.url
schema.registry.ssl.truststore.location
schema.registry.ssl.truststore.password

See:  https://docs.confluent.io/platform/current/schema-registry/security/index.html#clients-to-sr

On the client, configure the environment variables to set the SSL keystore and truststore in one of two ways:

(Recommended) Use the Schema Registry dedicated properties to configure the client:
schema.registry.ssl.truststore.location=/etc/kafka/secrets/kafka.client.truststore.jks
schema.registry.ssl.truststore.password=<password>
schema.registry.ssl.keystore.location=/etc/kafka/secrets/kafka.client.keystore.jks
schema.registry.ssl.keystore.password=<password>
schema.registry.ssl.key.password=<password>