CODEWITHSUNDEEP
linuxbashxargs
George N.
George N.

Reputation: 13

exec bash script for each item in list

I have a text file with URLs

http://example.com/1
http://example.com/2

etc.

I have a bash script that takes the URL as $1 and works with it. I would like to automate it and I have tried with

cat urls.txt | xargs -P0 bash -c myscript.sh

but $1 comes up as empty.

Upvotes: 1

Views: 2281

Answers (3)

Ole Tange
Ole Tange

Reputation: 33725

With GNU Parallel it looks like this:

cat urls.txt | parallel -j0 ./myscript.sh

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Upvotes: 0

Roman Kiselenko
Roman Kiselenko

Reputation: 44370

I suggest you to use read with a while loop, here is an example:

#!/bin/bash

while read -r line      # read a line from file.
do
  echo "$line"
  ./myscript.sh "$line"             # pass a line to the script
done < urls.txt

Upvotes: 2

chepner
chepner

Reputation: 531798

You don't need -c (or cat):

xargs -P0 bash myscript.sh < urls.txt

-c takes a string argument to use as the command, for example,

$ bash -c 'echo foo'
foo

When using -c, the next argument after the command string is used as the value for $0, not $1:

$ bash -c 'echo Command: $0; echo Arg: $1' zeroth first
Command: zeroth
Arg: first

Upvotes: 1

Related Questions