CODEWITHSUNDEEP
apache-nifi
Paul Bendevis
Paul Bendevis

Reputation: 2621

Nifi Broken Pipe ExecuteStreamCommand

In Nifi, I'm running an ExecuteStreamCommand processor which is calling a python script and passing the flowFile to stdin (Ignore StdIN: False). I get constant errors from the processor even though the python script is executing correctly and is able to read from stdin. Why is this error appearing and how can I debug/suppress it?

Failed to write flow file to stdin due to Broken pipe: java.io.IOException: Broken pipe

enter image description here

Pandas is used to read from stdin:

df = pd.read_csv(sys.stdin, usecols=schema_map.keys(),
                     nrows=50, dtype=parse_schema(schema_map))

Upvotes: 1

Views: 4812

Answers (1)

keith.g
keith.g

Reputation: 940

This error shows up because NiFi failed to persist the flowfile state in order to be able to show it's history.

Does this error still show up when you generate a small test flowfile (reading off a very small csv file)? If it does it could be a case of a large file causing a timeout.

For reference, this is the line of code that is raising the exception.

I was able to solve this by ignoring stdin but in your case I guess you could write the data to a file and pass the path instead.

Upvotes: 2

Related Questions