CODEWITHSUNDEEP
javabashparallel-processingpipetee
Albatross
Albatross

Reputation: 690

How to tee multiple times to a single file and program

Background

I am having following command stored as a shell script in execution.sh

cat input_file | tee output_file | java program

I used ./execution.sh & successfully to read from input_file, store the data in single output_file and also send as input to java program.

Problem

I want to output data from input_file multiple times to output_file and also to java program.

e.g. Reading same input_file say 5 times in parallel and send the data to a single output_file and single java program.

Edit

Tried solution

execution.sh

{
 python2 readLines.py &
 python2 readLines.py &
 python2 readLines.py &
 python2 readLines.py &
 python2 readLines.py &
} | tee  output_file | java program 

readLines.py

with open('inputfile') as f:
    for line in f:
       print line

I am using this currently, please comment if any issues anybody sees such as race condition etc here.

Upvotes: 1

Views: 961

Answers (1)

Socowi
Socowi

Reputation: 27275

Reading same input_file say 5 times in parallel and send the data to a single output_file and single java program

Let's just ignore the »parallel« part. Writes should only be made sequentially.

( for i in {1..5}; do cat input_file; done ) | tee out_file | java program

or in short

cat input_file{,,,,} | tee out_file | java program

Both commands print the file 5 times in a row.

If you really want to write in parallel, you could start the cat commands as background jobs: 

( for i in {1..5}; do cat input_file & done ) | tee out_file | java program

This method guarantess that output_file contained all bytes from input_file exactly five times, but (of course) interleaved. There is a high chance that not only the lines, but also the bytes will end up interleaved. What does that mean?

If you have the file

abc
xyz

and print it two times in parallel, the output might become

ababcc

xxyz
yz

If that doesn't bother you, also keep in mind that that there are sequences of bytes which lose/change their meaning if they aren't appearing in that sequence, for instance windows new lines \r\n or multibyte unicode characters.

Upvotes: 2

Related Questions