in nifi, how to call an external program that ask an input file and an output file in parameters

Question

I have an external program, an ebook converter, to convert .epub to .txt. This converter requests a file as input and another file as output, the filenames are important here because the extension is used to determine which conversion should be made, also from what I saw while testing, the program is performing seeks on the input file. So those contrains prenvents usage of named pipes or SDTIN redirections.

For another project, or at least a POC, I'll have to encapsulated an existing bundle of tools that are working the same way as above and recreate the workflows bettween in Nifi.

In short term, writing a custom processor is not possible.

So how should i do it ?

here are a couple of possible solutions I found :

• create a PutFile processor who will put the file in a temporary location, chain it with an executeStreamCommand who will execute the external command and put the output in a temporary location and chain it with a fetchFileProcessor. But the issue here may be that I have to found a way to clean the temporary files.

• another solution would be to create a script to use with ExecuteScript to do something like : write the flowfile to disk with filename based on attributes, executute the external command, read the outputfile to the flowfile and perform some cleaning. but from what I found, it's not that easy to write to disk from this processor, right ?

so what direction should I go, any advices ?

Answer 1

You are on the right track. I think you could do a sequence of processors like the following:

1. UpdateAttribute - create attributes for the input and output file names.
2. PutFile - write the temp input file.
3. ExecuteProcess - run the conversion utility. I recommend wrapping this in a shell script, which gives you an opportunity to clean up the temp input file when it completes.
4. FetchFile - read the conversion output into the FlowFile content. Use Completion Strategy of Delete File to clean up the converted file.