CODEWITHSUNDEEP
pythonshellparallel-processinggnu-parallel
midox
midox

Reputation: 17

Launch python commands in parallel using shell

I have a command in python this way:

Python my_prog in1.fa ins.fa out1.fa
Python my_prog in2.fa ins.fa out2.fa
Python my_prog in3.fa ins.fa out3.fa

I used the parallel command of GNU parallel and I assemble the files in1.fa, in2.fa and in3.fa in the one file IN.fa. My problem is I do not know how to put another agument or more in the parallel command. here is my command:

cat IN.fa | parallel -j 20 --cat --pipe --block 3M --recstart '>' time python my_prog.py

How can I make several arguments in the command Parallel please?

Upvotes: 0

Views: 122

Answers (1)

Ole Tange
Ole Tange

Reputation: 33685

Let us assume that my_prog can read from stdin and send output to stdout and that it takes a single argument (ins.fa):

parallel --pipepart -a in.fa --block 3M Python my_prog ins.fa > out.fa

If my_prog cannot read from stdin, but from a named pipe (fifo) this will work:

parallel --fifo --pipepart -a in.fa Python my_prog {} ins.fa > out.fa

If my_prog cannot read from a fifo, but only an actual file, this will work:

parallel --cat --pipepart -a in.fa Python my_prog {} ins.fa > out.fa

If my_prog cannot output to stdout, but can output to a fifo you can often use:

parallel --cat --pipepart -a in.fa Python my_prog {} ins.fa {#}.out /dev/stdout  > out.fa

Or:

parallel --cat --pipepart -a in.fa Python my_prog {} ins.fa {#}.out '>(cat)' > out.fa

If my_prog cannot output to a fifo, you need to have it output to a uniquely named file, which you can then cat and remove. Here we use the sequence number to make a unique file.

parallel --cat --pipepart -a in.fa Python my_prog {} ins.fa {#}.out '; cat {#}.out; rm {#}.out'  > out.fa

You really should consider walking through the tutorial. It will answer this and so many other questions: man parallel_tutorial

Upvotes: 1

Related Questions