CODEWITHSUNDEEP
gnu-parallel
mmdanziger
mmdanziger

Reputation: 4658

bash gnu parallel argfile syntax

I just discovered GNU parallel and I'm having some trouble running a simple parallel task. I have a simulation running over multiple values and I'd like to split it up to run in parallel using command line args. From the docs , it seems you can run parallel mycommand :::: myargfile in which myargfile contains the various arguments you would like to feed your command, in parallel. However, I didn't see any information on how the args should be listed and assumed a myargfile like this would work:

--pmin 0 --pmax 0.1
--pmin 0.1 --pmax 0.2
...

mycommand --pmin 0 --pmax 0.1 executes no problem. But when I run parallel mycommand :::: myargfile I get error: unknown option pmin 0 --pmax 0.1 (caught and decoded courtesy boost program options). parallel echo :::: myargfile correctly prints out the arguments. It's as if they are being wrapped in a string which the program can't read and not fed like they are from a standard bash script.

What's going on? How can I make this work?

Upvotes: 2

Views: 649

Answers (2)

Oto Brglez
Oto Brglez

Reputation: 4193

After digging through manual and help pages I came up with this example. Perhaps it will save someone out there. :)

#!/usr/bin/env bash

COMMANDS=(
  "cnn -a mode=flat"
  "cnn -a mode=xxx"
  "cnn_x -a mode=extreme"
)

parallel --verbose --progress --colsep ' ' scrapy crawl {.} ::: "${COMMANDS[@]}"

Upvotes: 0

mmdanziger
mmdanziger

Reputation: 4658

Following @DmitriChubarov's link to https://stackoverflow.com/a/6258206/1328439 , I discovered that I was lacking the colsep flag:

parallel --colsep ' ' mycommand :::: myargfile

successfully executes.

Upvotes: 2

Related Questions