Upgrading to Spring Cloud Stream debezium-source-kafka to version 5.0.0. results to no messages sent
Article ID: 373138
Updated On:
Products
Issue/Introduction
Using Spring Cloud Data Flow you used on of the debezium stream applications and followed example found here. You are able to send a message to kafka if you use/upgrade to debezium-source-kafka version 5.0.0
Looking into the application logs, you can see it was started successfully and also provides info on connector configuration
2024-07-23T03:35:04.154Z INFO 1 --- [debezium-source] [ main] o.s.i.d.inbound.DebeziumMessageProducer : started bean 'debeziumPublisher.debezium:inbound-channel-adapter#0'; defined in: 'class path resource [org/springframework/cloud/fn/supplier/debezium/DebeziumReactiveConsumerConfiguration.class]'; from source: 'bean method debeziumPublisher'
...
2024-07-23T03:35:04.655Z INFO 1 --- [debezium-source] [ main] s.a.d.s.k.DebeziumSourceKafkaApplication : Started DebeziumSourceKafkaApplication in 56.104 seconds (process running for 58.661)
2024-07-23T03:35:04.853Z INFO 1 --- [debezium-source] [pter#0-thread-1] d.DebeziumEngineBuilderAutoConfiguration : Debezium Engine completed with success:false, message:Connector configuration is not valid. The 'topic.prefix' value is invalid: A value is required
2024-07-23T03:36:35.271Z INFO 1 --- [debezium-source] [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
Environment
Spring Cloud DataFlow Server 2.11.0
cdc-debezium-source-kafka version 3.2.1 upgraded to debezium-source-kafka version 5.0.0
Cause
The issue is due to a invalid connector configuration on this particular error message 'topic.prefix' as of date of writing default does not have a default value and you need to put a value
2024-07-23T03:35:04.853Z INFO 1 --- [debezium-source] [pter#0-thread-1] d.DebeziumEngineBuilderAutoConfiguration : Debezium Engine completed with success:false, message:Connector configuration is not valid. The 'topic.prefix' value is invalid: A value is required
Resolution
To resolve this issue add debezium.properties.topic.prefix and assigned to a value to your configuration. Sample configuration below:
debezium.properties.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.properties.schema.history.internal=io.debezium.relational.history.MemorySchemaHistory
debezium.properties.offset.storage=org.apache.kafka.connect.storage.MemoryOffsetBackingStore
debezium.properties.topic.prefix=assign-topic-prefixvalue-here
debezium.properties.name=connector-name
debezium.properties.database.server.id=#####
debezium.properties.database.user=######
debezium.properties.database.password=########
debezium.properties.database..dbname=#######
debezium.properties.database.hostname=#######
debezium.properties.database.port=######
debezium.properties.schema=true
debezium.properties.key.converter.schemas.enable=true
debezium.properties.value.converter.schemas.enable=true
debezium.properties.transforms=unwrap
debezium.properties.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
debezium.properties.transforms.unwrap.add.fields=name,db
debezium.properties.transforms.unwrap.delete.handling.mode=none
debezium.properties.transforms.unwrap.drop.tombstones=true
Feedback
