CODEWITHSUNDEEP
phpparallel-processingweb-crawlergnu-parallel
Aryabhatt
Aryabhatt

Reputation: 657

GNU Parallel to run from a webpage

I am working on a demo and using GNU Parallel to run my web crawlers in parallel to help reduce the overall time. I am running it from a php file as follows:

<?php

   ....
   exec(" parallel -j 8 < commands.txt", $output);
   ....
?>

When I run this file from terminal, it works fine and does all the scraping work given in the commands.txt file, but the moment I try running it from a webpage (which is calling this php file), parallel gives me the following error:

parallel: Warning: $SHELL not set. Using /bin/sh.
Use of uninitialized value $ENV{"HOME"} in concatenation (.) or string at  /usr/bin/parallel line 943.
Use of uninitialized value $ENV{"HOME"} in concatenation (.) or string at   /usr/bin/parallel line 943.
Use of uninitialized value $ENV{"HOME"} in concatenation (.) or string at /usr/bin/parallel line 2207.
Use of uninitialized value $ENV{"HOME"} in concatenation (.) or string at /usr/bin/parallel line 2207.

Has anyone faced a similar problem and can help me with a solution to this problem ?

Thanks in advance.

Upvotes: 2

Views: 1349

Answers (1)

Ole Tange
Ole Tange

Reputation: 33740

GNU Parallel uses $HOME and $SHELL. $SHELL defaults to /bin/sh, but there is no default for $HOME. So set $HOME:

putenv('HOME=/tmp'); # Or some other place

You are expected to have write permissions in $HOME.

If you want to avoid the warning set $SHELL, too:

putenv('SHELL=/bin/sh');

Upvotes: 3

Related Questions