CODEWITHSUNDEEP
joinconcatenationapache-nifi
Micro_Andy
Micro_Andy

Reputation: 113

Nifi: How to concatenate flowfile to already existing tables in a directory?

This is a question about Nifi.

I made Nifi pipeline to convert flowfile with xml format to csv format. Now, I would like to concatenate or union the converted csv flowfile to existing tables by filename (which stands for table name as well).

Simply put, my processor flow is following.

  1. GetFile (from a particular directory) -> 2. Convert xml to csv -> 3.Update the flowfile with table name -> 4. PutFile (to a different directory)

But, at the end of the flow, PutFile processor throws an error, saying "file with the same name already exists".

I have no ideas how flowfile can be added to existing csv table. Any advice, tips, ideas are appreciated.

Thank you in advance.

Upvotes: 0

Views: 1023

Answers (1)

daggett
daggett

Reputation: 28599

there is no support to append file however you could use ExecuteGroovyScript to do it: 

def ff=session.get()
if(!ff)return
ff.read().withStream{s->
    String path = "./out_folder/${ff.filename}"
    //sync on file path to avoid conflict on same file writing (hope)
    synchronized(path){  
        new File( path ).append(s)
    }
}
REL_SUCCESS << ff

if you need to work with text (reader) content rather then byte (stream) content

the following example shows how to exclude 1 header line from flow file if destination file already exists

def ff=session.get()
if(!ff)return
ff.read().withReader("UTF-8"){r->
    String path = "./.data/${ff.filename}"
    //sync on file path to avoid conflict on same file writing (hope)
    synchronized(path){
        def fout = new File( path )
        if(fout.exists())r.readLine() //skip 1 line (header) only if out file already exists
        fout.append(r) //append to the file the rest of reader content
    }
}
REL_SUCCESS << ff

Upvotes: 2

Related Questions