CODEWITHSUNDEEP
hdfsapache-nifi
Adithya Sajjanam
Adithya Sajjanam

Reputation: 53

Nifi List File from multiple paths and route them to their respective destinations

I have 3 files at different locations for which I am using 3 list file processors and then the fetch file processor. Now I want to route these files to their respective destinations which is different for all 3 files. need some help on how to do this.

Thanks

Upvotes: 3

Views: 3525

Answers (2)

Vinayak Mishra
Vinayak Mishra

Reputation: 351

You can use simple if else statements and set destination on a flowfile attribute, resulting you can do the same in single processors. You can further tune them by number of threads on a single processor for a clean layout. PFB example assume hostname1 puts file to hostnameX/path1 and hostname2 puts file to hostnameY/path2 :

your new attribute for putfile path = ${sftp.remote.hostname:equals('hostname1'):ifElse('hostnameX/path1','hostnameY/path2')}

You can further nest ifElse in case of n number of hostnames and their corresponding destinations and use your new attribute as destination.

Upvotes: 0

Sivaprasanna Sethuraman
Sivaprasanna Sethuraman

Reputation: 4132

As per the routing logic you had mentioned above, you could have the flow as :

ListFile(s) -> FetchFile(s) -> RouteOnAttribute -> PutFile(s)

In RouteOnAttribute, you could use NiFi Expression Language support. For example, you can have three attributes:

destination.one : ${filename:startsWith('abc')}
destination.two : ${filename:startsWith('def')}
destination.three : ${filename:startsWith('xyz')}

Explanation

You will have three ListFile processors connected to three FetchFile processors. Then connect the Success output of all three processors to RouteOnAttribute, and after defining the above three relationships, you can connect the three relations to the relevant PutFile processors.

In the above example, the RouteOnAttribute routes the files whose name starts with abc to the relationship destination.one, filenames that begin with def goes to destination.two and so on.

enter image description here

enter image description here

Upvotes: 6

Related Questions