CODEWITHSUNDEEP
jsoncsvapache-nifi
brandon lee
brandon lee

Reputation: 99

NiFi How to MergeContent into a one row csv

I am new to using Nifi and am having a problem merging two csv files into a single row csv. I apologize if this question has been answered below. But, I have tried looking all over the internet but was unable to find a solution specific to my need.

So here is the problem: I have data that is in Json format and I need to transform the data into csv and the file output should be a flat csv file.

I start with a Json Data that looks like this:

{"header": {"messageId": "xxx", "name":"xxx",..}, "content": {"TimeStamp":"xxx",..}}

I first split the json message into its header and content components and use ConvertRecord to convert the data from json to csv. I have it figured out upto this part.

The two csv records look like this:

content section from json to csv

header section from json to csv

Then I use the MergeContent processor to merge the two csv data, to hopefully get a single csv data with the header and content data in one row (with the header being in row 1 and the values in row 2). However, on merging the contents I get this merge results from MergeContent processor

As you can see, the headers from the 'header' header section is in line 1 and the headers from the 'content' section is in line 3 and the values are in line 2 and 4 respectively. But, what I want is both the headers of the 'header' section and 'content' section to be in line 1 and the values of both sections to be in line 2.

Is my approach to solving the problem incorrect? Is there a better approach to this problem? Also, what can I do to correct this current approach to get my desired result?

Any help will be appreciated. Thank you in advance.

Upvotes: 1

Views: 454

Answers (1)

Kolban
Kolban

Reputation: 15266

Maybe a different approach. Imagine we had JSON that looked like:

{
  "part1": {
    "A": "A1",
    "B": "B1"
  },
  "part2": {
    "X": "X1",
    "Y": "Y1"
  }
}

and what you wanted was CSV that looked like:

A,B,X,Y
A1,B1,X1,Y1

then we could solve this puzzle by reworking the original JSON to a new JSON description that looks like:

{
  "A":"A1",
  "B":"B1",
  "X":"X1",
  "Y":"Y1"
}

we can then use our JSON to CSV converter which should produce the format you are looking for.

To transform the original JSON to our desired JSON we could use the JoltTransformJSON processor. An example specification might be:

[{
    "operation": "shift",
    "spec": {
        "part1": {
          "*": "&"
        },
        "part2": {
          "*": "&"
        }
    }
}]

Upvotes: 1

Related Questions