CODEWITHSUNDEEP
bashgnu-parallel
Christophe Marchand
Christophe Marchand

Reputation: 91

gnu parallel processing multiple files on various instances

I want to process many files on many database instances. Instances are on various computers, one computer may host up to 4 instances.

A process is run on an instance via a bash shell :

process-on.sh <file_to_process> <instance_name>

process-on.sh knows how to access the instance (server, script-name, and so on) from the instance_name. The same unix client is used to run processes for all instances on one host, just the remote script name varies.

I would like instances act as a pool of processors, and want to use gnu-parallel as a distributor of files between instances.

I would like to use gnu-parallel, but it is mainly to distribute between computers, not isntances... Maybe there is another tool which can work with pools...

Any hints will be much appreciated.

Best, Christophe

Upvotes: 1

Views: 151

Answers (1)

Ole Tange
Ole Tange

Reputation: 33740

You will need a way to convert a number into an instance - maybe some sort of lookup table.

instances=13
parallel -j$instances 'sleep 0.$RANDOM; echo this is file {} being processed on instance {%}' ::: {1..100}

Upvotes: 0

Related Questions