CODEWITHSUNDEEP
apache-nifi
gashu
gashu

Reputation: 873

Nifi - Processing Huge json array of records into single records

I am using a invoke HTTP processor that gives an array of JSON objects. the size of the response is between 2-3 GB. The response looks something like this - 

[
{
  "id": 17,
  "name": "ONE by AOL: Video"
},
{
  "id": 63,
  "name": "Adform"
} ---

Later at the downstream, I want to use each JSON object at a time because I need to apply some filters on it and save it in the database later. Want to know if splitjson is the right processor to go for? Currently, I am getting Out of memory exception while using the splitjson even when I have given 5GB in the bootstrap.conf file and also 2gb to the queue configs. I can go for more RAM if that's the only option?

Upvotes: 1

Views: 723

Answers (1)

Bryan Bende
Bryan Bende

Reputation: 18670

You should use the record processors to avoid needing to split. You can use QueryRecord for filtering and PutDatabaseRecord for inserting to a database.

Upvotes: 2

Related Questions