CODEWITHSUNDEEP
linuxbashshellunix
user10437665
user10437665

Reputation: 115

run multiple commands in parallel in unix

I have few files and I have to cut few columns from that files to generate new files unix. I tried to do it in loop as selecting files in directory and generating new files but as directory having 100 such files it takes lot of time to generate new files.

Can anyone please help if I can select 10 files in parallel and generate 10 new files and again next set of 10 files as it will reduce the time.

i need sample unix code block for this 

cut -b 1-10,25-50,65-79 file1.txt > file_cut1.txt

cut -b 1-10,25-50,65-79 file2.txt > file_cut2.txt

Upvotes: 1

Views: 1329

Answers (1)

Mark Setchell
Mark Setchell

Reputation: 208077

You can do that quite simply with GNU Parallel like this:

parallel 'cut -b 1-10,25-50,65-79 {} > {.}_cut.txt' ::: file*txt

where:

  • {} represents the current filename, and
  • {.} represents the current filename without its extension.

Make a backup of the files in your directory before trying this, or any unfamiliar commands.

It will process your files in parallel, doing N at a time, where N is the number of cores in your CPU. If you want it to do, say 8, jobs at a time, use:

parallel -j 8 ...

If you want to see what it would do, without actually doing anything, use:

parallel --dry-run ...

Upvotes: 3

Related Questions