CODEWITHSUNDEEP
linuxbashwebcurl
Datenkralle
Datenkralle

Reputation: 45

Download multiple files simultaneously with variable in URL with Curl

So I have a little bash script:

for store in {4..80}
do
  for page in {1..200} 
  do
    curl 'https://website.com/shop.php?store_id='$store'&page='$page'&PHPSESSID=sessionID' -O
  done
done

The script is working but downloads all 200 store pages of all stores from 4-80 one after another, which takes a lot of time. (Notice the bash variable store and page in the curl URL)

My goal would be to run as many curl requests simultaneously for each store/page instead getting it worked on one after one, to save time.

Is this possible?

Upvotes: 2

Views: 1692

Answers (2)

Antonio Petricca
Antonio Petricca

Reputation: 11050

Try changing you script as follow:

for store in {4..80}; do
  for page in {1..200}; do
    curl 'https://website.com/shop.php?store_id='$store'&page='$page'&PHPSESSID=sessionID' -O &
  done
done

# Wait for spawned cURLs...
wait

Upvotes: 1

Cyrus
Cyrus

Reputation: 88939

curl can run loops itself. This limits curl to 100 simultaneously sessions:

curl 'https://website.com/shop.php?store_id=[4-80]&page=[1-200]&PHPSESSID=sessionID' -O --parallel --parallel-max 100

Upvotes: 7

Related Questions