CODEWITHSUNDEEP
csvapache-nifidata-processing
Himsara Gallege
Himsara Gallege

Reputation: 945

Apache NiFi: Identifying csv records containing special characters

Using Apache NiFi I need to filter out the records in a csv which have a set of special characters.

As an example if the set of special characters are "FFF". My csv would be

name,age,city
John,23,New York
FFF,45,London
Himsara,18,Adelaide

Then the second record must be taken out from the csv and put into another csv. Also even if "FFF" is in the city or age columns the whole record must be removed.

Please suggest me the processors that are needed for me to achieve this. Also it would be really helpful if u can list out the configurations, that are needed to be changed.

Upvotes: 0

Views: 1119

Answers (2)

DarkLeafyGreen
DarkLeafyGreen

Reputation: 70416

As an alternative, you can use the RouteText processor. It will split the flow file based on a condition. Lines containing FFF will route to the matched relationship, the other lines will route to the unmatched relationship.

enter image description here

RouteText processor setup:

settings

Upvotes: 2

notNull
notNull

Reputation: 31490

Use QueryRecord processor in nifi and define Record Reader/Writer Avro schemas to read your incoming flowfile.

Then add new property to QueryRecord processor as (Apache calcite sql)

select * from FLOWFILE where name !="FFF"

Now use the newly added relationship from QueryRecord processor for further processing and NiFi will result flowfile where name is not equal to 'FFF'.

Upvotes: 2

Related Questions