CODEWITHSUNDEEP
bashxargs
Amir
Amir

Reputation: 2258

Multithreading xargs with input from cat

I have a text file files.txt on my server, on each line is a file with the full path, e.g. /home/lelouch/dir/randomfile.txt.

I want to loop through files.txt, and pass each filename to another script.

I have gotten this to work like this:

cat /home/lelouch/dir/files.txt | xargs -0 -n 1 -P 30 /home/lelouch/bin/script.

The problem is, although I want to process it 30 files at a time, it's only happening 1 at a time. I've tried a few other ways, but I haven't gotten it to work like I want.

Any ideas?

Upvotes: 6

Views: 10766

Answers (3)

Olathe
Olathe

Reputation: 1895

You say that each line is a filepath, but you use the -0 option of xargs, which switches the separator from a newline to a null character. From the man page:

Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (every character is taken literally)....

Don't use the -0 option:

cat /home/lelouch/dir/files.txt | xargs -P 30 -n 1 /home/lelouch/bin/script

Upvotes: 8

Ade YU
Ade YU

Reputation: 2362

--max-args is the right option you need

cat /home/lelouch/dir/files.txt | xargs --max-args=30 /home/lelouch/bin/script

Upvotes: 0

John Zwinck
John Zwinck

Reputation: 249394

I think you want GNU Parallel.

Upvotes: 3

Related Questions