CODEWITHSUNDEEP
avroapache-nifiparquet
koala421
koala421

Reputation: 816

Apache NiFi not converting recognizing decimal type in convertJsontoAvro Processor

I have a ConvertJsontoAvro processor in NiFi 1.4 and am having difficulty getting the proper datatype of decimal within the avro. The data is being transformed into bytes using logical Avro data types within ExecuteSQL processor, converting avro to Json using ConvertAvrotoJSON processor, and then using ConvertJsonToAvro processor to put into HDFS using PutParquet.

My schema is :

{ "type" : "record", "name" : "schema", "fields" : [ { "name" : "entryDate", "type" : [ "null", { "type" : "long", "logicalType" : "timestamp-micros" } ], "default" : null }, { "name" : "points", "type" : [ "null", { "type" : "bytes", "logicalType" : "decimal", "precision" : 18, "scale" : 6 } ], "default" : null }] }

My JSON:

{ "entryDate" : 2018-01-26T13:48:22.087, "points" : 6.000000 }

I get an error for the avro saying

Cannont convert field points: Cannot resolve union : {"bytes": "+|Ð" not in ["null", {"type":"bytes","logicalType":"decimal","precision":18,"scale":6}]"

Is there some type of work around for this?...

Upvotes: 1

Views: 2558

Answers (2)

RenanSS
RenanSS

Reputation: 131

For anyone interested, I was able to set the decimal and a default value as null (in cases when the field is null or missing), currently using Nifi 1.14.0

{
  "name": "value",
  "type": [
    "null",
    {
      "type": "bytes",
      "logicalType": "decimal",
      "precision": 8,
      "scale": 4
    }
  ],
  "default": null
}

Upvotes: 1

bartektartanus
bartektartanus

Reputation: 16080

Currently you cannot mix null type and logical types due to bug in Avro. Check this still unresolved issue: https://issues.apache.org/jira/browse/AVRO-1891

Also the defaults value cannot be null. This should work for you:

{
    "type" : "record",
    "name" : "schema",
    "fields" : [ {
      "name" : "entryDate",
      "type" : {
        "type" : "long",
        "logicalType" : "timestamp-micros"
      },
      "default" : 0
    }, {
      "name" : "points",
      "type" : {
        "type" : "bytes",
        "logicalType" : "decimal",
        "precision" : 18,
        "scale" : 6
      },
      "default" : ""
    }]
  }

Upvotes: 1

Related Questions