Using the built in Avro serializer causes AvroTypeException in DevTest Kafka step
Article ID: 246579
Updated On:
Products
Issue/Introduction
While testing the built in Avro serializer with a particular schema in the DevTest Kafka step (instead of using the Confluent Avro serializer),
the following exception was encountered:
============================================================================
| Expected start-union. Got VALUE_STRING
============================================================================
| Step: Kafka SEND
----------------------------------------------------------------------------
| Message: Expected start-union. Got VALUE_STRING
----------------------------------------------------------------------------
| Trapped Exception: Expected start-union. Got VALUE_STRING
| Trapped Message: org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING
----------------------------------------------------------------------------
STACK TRACE
org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING
at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:511)
at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:430)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:282)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at com.itko.lisa.messaging.providers.kafka.serializer.AvroUtils$AvroSerializer.serialize(AvroUtils.java:79)
at com.itko.lisa.messaging.providers.kafka.serializer.AvroUtils$AvroSerializer.serialize(AvroUtils.java:57)
at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:929)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:889)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:775)
at com.itko.lisa.messaging.providers.kafka.asset.KafkaUtils.sendSingleMessage(KafkaUtils.java:348)
at com.itko.lisa.messaging.providers.kafka.operation.KafkaSendOperation.doExecute2(KafkaSendOperation.java:308)
at com.itko.lisa.messaging.providers.kafka.operation.KafkaSendOperation.doExecute2(KafkaSendOperation.java:48)
at com.itko.lisa.asset.operation.AbstractRetryableRuntimeOperation.doExecute(AbstractRetryableRuntimeOperation.java:30)
at com.itko.lisa.asset.operation.AbstractRuntimeOperation.execute(AbstractRuntimeOperation.java:167)
at com.itko.lisa.messaging.builder.AbstractSendReceiveOperation.doExecute(AbstractSendReceiveOperation.java:156)
at com.itko.lisa.asset.operation.AbstractRuntimeOperation.execute(AbstractRuntimeOperation.java:167)
at com.itko.lisa.asset.operation.AbstractOperationStep.execute(AbstractOperationStep.java:176)
at com.itko.lisa.asset.operation.AbstractOperationStep.execute(AbstractOperationStep.java:125)
at com.itko.lisa.test.TestNode.executeNode(TestNode.java:995)
at com.itko.lisa.test.TestCase.execute(TestCase.java:1297)
at com.itko.lisa.test.TestCase.execute(TestCase.java:1198)
at com.itko.lisa.test.TestCase.executeNextNode(TestCase.java:1183)
at com.itko.lisa.editor.WalkThruPanel.prepAndExecNode(WalkThruPanel.java:1120)
at com.itko.lisa.editor.WalkThruPanel.access$900(WalkThruPanel.java:73)
at com.itko.lisa.editor.WalkThruPanel$10.doCallback(WalkThruPanel.java:1022)
at com.itko.util.swing.panels.ProcessingDialog$2.run(ProcessingDialog.java:195)
at java.lang.Thread.run(Thread.java:748)
============================================================================
Example JSON message:
{
"header":{
"corrId":"ea1bd730-d1a3-4e3b-964f-03541c9b1234",
"timestamp":"2022-06-30T09:49:17.067Z"
},
"evId":"ea1bd730-d1a3-4e3b-964f-03541c9b1234",
"evSummary":"eventSummary",
}
============================================================================
Example schema:
{
"type": "record",
"name": "name",
"namespace": "namespace",
"doc": "doc",
"fields": [
{
"name": "header",
"type": {
"type": "record",
"name": "name",
"namespace": "namespace",
"fields": [
{
"name": "corrid",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "doc"
},
{
"name": "timestamp",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "doc"
}
]
}
},
{
"name": "evId",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "evId"
},
{
"name": "evSummary",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "doc"
}
],
"semVer": "1.0.0"
}
Environment
All supported DevTest releases.
Component : DevTest Workstation
Resolution
The standard avro tool has the same error:
java -jar avro-tools-1.7.7.jar fromjson --schema-file sample-schema.avsc sample-message-to-send.JSON > outcome.json
Refer to these articles:
Apache Avro Specification JSON encoding
How to fix Expected start-union. Got VALUE_NUMBER_INT when converting JSON to Avro on the command line
Change the message as follows:
{
"header":{
"corrId":"ea1bd730-d1a3-4e3b-964f-03541c9b1234",
"timestamp":"2022-06-30T09:49:17.067Z"
},
"evId":{"string":"ea1bd730-d1a3-4e3b-964f-03541c9b1234"},
"evSummary":{"string":"eventSummary"}
}
Another option is to update the schema and remove the null:
From:
"name": "eventId",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
To:
"name": "eventId",
"type":
{
"type": "string",
"avro.java.string": "String"
}
,
Feedback
