Using the built in Avro serializer causes AvroTypeException in DevTest Kafka step
search cancel

Using the built in Avro serializer causes AvroTypeException in DevTest Kafka step

book

Article ID: 246579

calendar_today

Updated On:

Products

Service Virtualization

Issue/Introduction

While testing the built in Avro serializer with a particular schema in the DevTest Kafka step (instead of using the Confluent Avro serializer),
the following exception was encountered:

============================================================================
| Expected start-union. Got VALUE_STRING
============================================================================
| Step:        Kafka SEND
----------------------------------------------------------------------------
| Message:     Expected start-union. Got VALUE_STRING
----------------------------------------------------------------------------
| Trapped Exception: Expected start-union. Got VALUE_STRING
| Trapped Message:   org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING
----------------------------------------------------------------------------
STACK TRACE
org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING
 at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:511)
 at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:430)
 at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:282)
 at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
 at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
 at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
 at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
 at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
 at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
 at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
 at com.itko.lisa.messaging.providers.kafka.serializer.AvroUtils$AvroSerializer.serialize(AvroUtils.java:79)
 at com.itko.lisa.messaging.providers.kafka.serializer.AvroUtils$AvroSerializer.serialize(AvroUtils.java:57)
 at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
 at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:929)
 at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:889)
 at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:775)
 at com.itko.lisa.messaging.providers.kafka.asset.KafkaUtils.sendSingleMessage(KafkaUtils.java:348)
 at com.itko.lisa.messaging.providers.kafka.operation.KafkaSendOperation.doExecute2(KafkaSendOperation.java:308)
 at com.itko.lisa.messaging.providers.kafka.operation.KafkaSendOperation.doExecute2(KafkaSendOperation.java:48)
 at com.itko.lisa.asset.operation.AbstractRetryableRuntimeOperation.doExecute(AbstractRetryableRuntimeOperation.java:30)
 at com.itko.lisa.asset.operation.AbstractRuntimeOperation.execute(AbstractRuntimeOperation.java:167)
 at com.itko.lisa.messaging.builder.AbstractSendReceiveOperation.doExecute(AbstractSendReceiveOperation.java:156)
 at com.itko.lisa.asset.operation.AbstractRuntimeOperation.execute(AbstractRuntimeOperation.java:167)
 at com.itko.lisa.asset.operation.AbstractOperationStep.execute(AbstractOperationStep.java:176)
 at com.itko.lisa.asset.operation.AbstractOperationStep.execute(AbstractOperationStep.java:125)
 at com.itko.lisa.test.TestNode.executeNode(TestNode.java:995)
 at com.itko.lisa.test.TestCase.execute(TestCase.java:1297)
 at com.itko.lisa.test.TestCase.execute(TestCase.java:1198)
 at com.itko.lisa.test.TestCase.executeNextNode(TestCase.java:1183)
 at com.itko.lisa.editor.WalkThruPanel.prepAndExecNode(WalkThruPanel.java:1120)
 at com.itko.lisa.editor.WalkThruPanel.access$900(WalkThruPanel.java:73)
 at com.itko.lisa.editor.WalkThruPanel$10.doCallback(WalkThruPanel.java:1022)
 at com.itko.util.swing.panels.ProcessingDialog$2.run(ProcessingDialog.java:195)
 at java.lang.Thread.run(Thread.java:748)
============================================================================

Example JSON message:
{
   "header":{
      "corrId":"ea1bd730-d1a3-4e3b-964f-03541c9b1234",
      "timestamp":"2022-06-30T09:49:17.067Z"
   },
   "evId":"ea1bd730-d1a3-4e3b-964f-03541c9b1234",
   "evSummary":"eventSummary",
}

============================================================================

Example schema: 
{
  "type": "record",
  "name": "name",
  "namespace": "namespace",
  "doc": "doc",
  "fields": [
    {
      "name": "header",
      "type": {
        "type": "record",
        "name": "name",
        "namespace": "namespace",
        "fields": [
          {
            "name": "corrid",
            "type": {
              "type": "string",
              "avro.java.string": "String"
            },
            "doc": "doc"
          },
          {
            "name": "timestamp",
            "type": {
              "type": "string",
              "avro.java.string": "String"
            },
            "doc": "doc"
          }
        ]
      }
    },
    {
      "name": "evId",
      "type": [
        "null",
        {
          "type": "string",
          "avro.java.string": "String"
        }
      ],
      "doc": "evId"
    },
    {
      "name": "evSummary",
      "type": [
        "null",
        {
          "type": "string",
          "avro.java.string": "String"
        }
      ],
      "doc": "doc"
    }
  ],
  "semVer": "1.0.0"
}

Environment

All supported DevTest releases.

Component : DevTest Workstation

Resolution

The standard avro tool has the same error:
java -jar avro-tools-1.7.7.jar fromjson --schema-file sample-schema.avsc sample-message-to-send.JSON > outcome.json

Refer to these articles:
Apache Avro Specification JSON encoding
How to fix Expected start-union. Got VALUE_NUMBER_INT when converting JSON to Avro on the command line

Change the message as follows:

{
"header":{
"corrId":"ea1bd730-d1a3-4e3b-964f-03541c9b1234",
"timestamp":"2022-06-30T09:49:17.067Z"
},
"evId":{"string":"ea1bd730-d1a3-4e3b-964f-03541c9b1234"},
"evSummary":{"string":"eventSummary"}
}

Another option is to update the schema and remove the null:
From:
      "name": "eventId",
      "type": [
        "null",
        {
          "type": "string",
          "avro.java.string": "String"
        }
      ],

To: 
      "name": "eventId",
      "type":   
        {
          "type": "string",
          "avro.java.string": "String"
        }
      ,