CODEWITHSUNDEEP
apache-nifi
septagram
septagram

Reputation: 465

Apache NiFi - Using multiple FlowFiles as input to a processor

I have a workflow where two or more inputs have set operations performed on them (union, complement, etc.) to produce a single output. I expect to have to write a processor to do the set logic myself, but is it even possible to work with multiple flowfiles of different provenance and work on them simultaneously?

Upvotes: 4

Views: 5972

Answers (1)

James
James

Reputation: 11931

NiFi processors can operate on all of the flowfiles in their input queue(s). For example:

@Override
public void onTrigger(final ProcessContext context, final ProcessSession session) {
    List<FlowFile> flowFiles = session.get(context.getProperty(BATCH_SIZE).asInteger());
    if (flowFiles == null || flowFiles.size() == 0) {
        return;
    }
    // process flowFiles
    ...

You can use the Funnel component to bring multiple inputs together into a single input queue, which can then share the same backpressure and prioritization settings.

NiFi Funnel consolidating input queues

Upvotes: 7

Related Questions