How to configure the 'Kafka Send Receive' step for SSL and the Confluent Avro Schema Registry over HTTPS.
All supported DevTest releases and platforms.
N/A
For access to the Schema Registry the Consumer in the Kafka step needs to be configured with the Confluent Kafka Avro Serializer and Deserializer:
Class Name: io.confluent.kafka.serializers.KafkaAvroDeserializer
The DevTest lib folder should have the external jars to implement this functionality.
For example if the server is running Confluent 7.0.1, the following jars should be copied from the Confluent Kafka server installation:
12/04/2021 12:19 PM 589,142 avro-1.10.1.jar
12/04/2021 10:14 AM 16,203,680 confluent-metrics-7.0.1-ce.jar
12/04/2021 12:19 PM 75,703 jackson-annotations-2.12.3.jar
12/04/2021 12:19 PM 365,220 jackson-core-2.12.3.jar
12/04/2021 12:19 PM 1,515,756 jackson-databind-2.12.3.jar
02/08/2022 02:55 PM 809,924 jdk8-2.5.2.jar
12/04/2021 12:19 PM 30,974 kafka-avro-serializer-7.0.1.jar
12/04/2021 10:14 AM 5,775,017 kafka-clients-7.0.1-ce.jar
12/04/2021 12:19 PM 23,334 kafka-json-schema-serializer-7.0.1.jar
12/04/2021 12:19 PM 10,108 kafka-json-serializer-7.0.1.jar
12/04/2021 12:19 PM 177,393 kafka-schema-registry-client-7.0.1.jar
12/04/2021 12:19 PM 32,248 kafka-schema-serializer-7.0.1.jar
DevTest Kafka step configuration for access to the topic over SSL is configured with the SSL context:
See: Section "Apache Kafka Assets" in the documentation of the DevTest release you are running.
DevTest Kafka step configuration for access to the Schema Registry over HTTPS is configured in the Consumer:
As the class and jars being used for the Deserializer are from Confluent, follow the Confluent documentation for the properties that can be used.
In this case:
schema.registry.url
schema.registry.ssl.truststore.location
schema.registry.ssl.truststore.password
See: https://docs.confluent.io/platform/current/schema-registry/security/index.html#clients-to-sr
On the client, configure the environment variables to set the SSL keystore and truststore in one of two ways:
(Recommended) Use the Schema Registry dedicated properties to configure the client:
schema.registry.ssl.truststore.location=/etc/kafka/secrets/kafka.client.truststore.jks
schema.registry.ssl.truststore.password=<password>
schema.registry.ssl.keystore.location=/etc/kafka/secrets/kafka.client.keystore.jks
schema.registry.ssl.keystore.password=<password>
schema.registry.ssl.key.password=<password>