CODEWITHSUNDEEP
jsonavroapache-nifi
Sergey Shcherbakov
Sergey Shcherbakov

Reputation: 4778

NiFi non-Avro JSON Reader/Writer

It appears that the standard Apache NiFi readers/writers can only parse JSON input based on Avro schema.

Avro schema is limiting for JSON, e.g. it does not allow valid JSON properties starting with digits.

JoltTransformJSON processor can help here (it doesn't impose Avro limitations to how the input JSON may look like), but it seems that this processor does not support batch FlowFiles. It is also not based on the readers and writers (maybe because of that).

Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form

{"myprop":"myval","12345":"12345",...}
{"myprop":"myval2","12345":"67890",...}

and transform it to other JSON structure, e.g. defined by JSON schema, and e.g. using JSON Patch transformation, without writing my own processor?

Update

I am using Apache NiFi 1.7.1

Update 2

Unfortunately, @Shu's suggestion did work. I am getting same error. Reduced the case to a single UpdateRecord processor that reads JSON with numeric properties and writes to a JSON without such properties using

myprop : /data/5836c846e4b0f28d05b40202

mapping. Still same error :(

enter code here

Upvotes: 0

Views: 635

Answers (1)

notNull
notNull

Reputation: 31470

it does not allow valid JSON properties starting with digits?

This bug NiFi-4612 fixed in NiFi-1.5 version, We can use AvroSchemaRegistry to defined your schema and change the

Validate Field Names

false

Then we can have avro schema field names starting with digits.

For more details refer to this link.

Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form?

This bug NiFi-4456 fixed in NiFi-1.7, if you are not using this version of NiFi then we can do workaround to create an array of json messages with ,(comma delimiter) by using.

Flow:

1.SplitText //split the flowfile with 1 line count
2.MergeRecord //merge the flowfiles into one
3.ConvertRecord

For more details regards to this particular issues refer to this link(i have explained with the flow).

Upvotes: 3

Related Questions