Kafka multinode cluster ssl_config.sh Health Check fail

Products

Network Observability

Issue/Introduction

When running the ssl_config.sh Health Check on a 3 node Kafka Cluster the Producer and Consumer checks fail

./ssl_config.sh

SSL Configuration
This utility allows you to check your SSL configuration and enable NetOps Kafka to use SSL.

1) SSL Health Check 3) Revert to Default Settings
2) Configure SSL 4) Quit

2025-07-28T15:51:37-04:00 SSL Configuration
2025-07-28T15:51:37-04:00 This utility allows you to check your SSL configuration and enable NetOps Kafka to use SSL.
2025-07-28T15:51:37-04:00
2025-07-28T15:51:38-04:00 Choice: SSL Health Check
2025-07-28T15:51:38-04:00 ================================================================================
2025-07-28T15:51:38-04:00 Performing SSL Health Check
2025-07-28T15:51:38-04:00 ================================================================================
~
~
2025-07-28T15:51:46-04:00 ============= Validating kafka/config/producer.properties ======================
2025-07-28T15:51:46-04:00 [ PASS ] security.protocol=SSL
2025-07-28T15:51:46-04:00 [ PASS ] ssl.keystore.location=/opt/CA/netops-kafka/ssl/keystore.pfx
2025-07-28T15:51:46-04:00 [ PASS ] ssl.keystore.password=*******
2025-07-28T15:51:46-04:00 [ PASS ] ssl.keystore.type=PKCS12
2025-07-28T15:51:46-04:00 [ PASS ] ssl.key.password=*******
2025-07-28T15:51:47-04:00 [ PASS ] ssl.keystore.password's password can access /opt/CA/netops-kafka/ssl/keystore.pfx
2025-07-28T15:51:47-04:00 [ PASS ] ssl.truststore.location=/opt/CA/netops-kafka/ssl/keystore.pfx
2025-07-28T15:51:47-04:00 [ PASS ] ssl.truststore.password=******
2025-07-28T15:51:47-04:00 [ PASS ] ssl.truststore.type=PKCS12
2025-07-28T15:51:47-04:00 [ PASS ] ssl.truststore.password's password can access /opt/CA/netops-kafka/ssl/keystore.pfx
2025-07-28T15:51:58-04:00 [ FAIL ] Producer connection to the broker failed
2025-07-28T15:51:58-04:00
2025-07-28T15:51:58-04:00 ============= Validating kafka/config/consumer.properties ======================
2025-07-28T15:51:58-04:00 [ PASS ] security.protocol=SSL
2025-07-28T15:51:58-04:00 [ PASS ] ssl.keystore.location=/opt/CA/netops-kafka/ssl/keystore.pfx
2025-07-28T15:51:58-04:00 [ PASS ] ssl.keystore.password=*******
2025-07-28T15:51:58-04:00 [ PASS ] ssl.keystore.type=PKCS12
2025-07-28T15:51:58-04:00 [ PASS ] ssl.key.password=*******
2025-07-28T15:51:58-04:00 [ PASS ] ssl.keystore.password's password can access /opt/CA/netops-kafka/ssl/keystore.pfx
2025-07-28T15:51:58-04:00 [ PASS ] ssl.truststore.location=/opt/CA/netops-kafka/ssl/keystore.pfx
2025-07-28T15:51:58-04:00 [ PASS ] ssl.truststore.password=*******
2025-07-28T15:51:58-04:00 [ PASS ] ssl.truststore.type=PKCS12
2025-07-28T15:51:59-04:00 [ PASS ] ssl.truststore.password's password can access /opt/CA/netops-kafka/ssl/keystore.pfx
2025-07-28T15:52:09-04:00 [ FAIL ] Consumer connection to the broker failed

Cause

Replication Factor was set too high.

Resolution

Lowered the replication factor from 3 to 2 and now the ssl_config.sh Health Check succeeds.

/opt/CA/netops-kafka/kafka/config/server.properties

# Replication Configuration
default.replication.factor=2 # Default replication factor for new topics

Additional Information

The replication factor error was discovered while trying to run the consumer perf test manually

cd /opt/CA/netops-kafka/kafka/bin/

./kafka-consumer-perf-test.sh --topic ssl_config_hc --bootstrap-server=<kafkaServerA>:9092,<kafkaServerB>:9092 --consumer.config=../config/consumer.properties --messages 5

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec

[2025-07-28 17:29:58,265] WARN [Consumer clientId=perf-consumer-client, groupId=perf-consumer-73500] The metadata response from the cluster reported a recoverable issue with correlation id 2 : {ssl_config_hc=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)