CODEWITHSUNDEEP
phpcurl
Fawzan
Fawzan

Reputation: 4849

Making multiple curl requests without timeout.

I am developing a system where I need to fetch 5000+ users location using multiple GET request. Unfortunately the API endpoint doesn't support multiple client ids. ie. I have to make 5000+ unique get requests to fetch their locations and use (the cumulative response) to make another API call.

I am using CURL to make the requests. I used the following snippet[1] to make the request. 

<?php

function multiRequest($data, $options = array()) {

  // array of curl handles
  $curly = array();
  // data to be returned
  $result = array();

  // multi handle
  $mh = curl_multi_init();

  // loop through $data and create curl handles
  // then add them to the multi-handle
  foreach ($data as $id => $d) {

    $curly[$id] = curl_init();

    $url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
    curl_setopt($curly[$id], CURLOPT_URL,            $url);
    curl_setopt($curly[$id], CURLOPT_HEADER,         0);
    curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);

    // post?
    if (is_array($d)) {
      if (!empty($d['post'])) {
        curl_setopt($curly[$id], CURLOPT_POST,       1);
        curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']);
      }
    }

    // extra options?
    if (!empty($options)) {
      curl_setopt_array($curly[$id], $options);
    }

    curl_multi_add_handle($mh, $curly[$id]);
  }

  // execute the handles
  $running = null;
  do {
    curl_multi_exec($mh, $running);
  } while($running > 0);


  // get content and remove handles
  foreach($curly as $id => $c) {
    $result[$id] = curl_multi_getcontent($c);
    curl_multi_remove_handle($mh, $c);
  }

  // all done
  curl_multi_close($mh);

  return $result;
}

?>

It works perfectly for small number of requests but when I try to hit 1000+ it gets timeout. 

    $data = [];

    for ($i = 0; $i < 1000; $i++) {
        $data[] = 'https://foo.bar/api/loc/v/queries/location?address=XXXXXXXXX';

    }
    $token = $this->refresh();

    $r = $this->multiRequest($data, $token);

What is the best approach to solve this issue?

Upvotes: 1

Views: 2177

Answers (1)

Vagharsh
Vagharsh

Reputation: 406

Is there a way to modify endpoint API to allow to process multiple ids? If yes, that is preferred, because if you run several thousands of requests at the same time, you actually making something like DDoS attack.

However, you may want to check PHP's curl_multi_* functions (https://www.php.net/manual/en/function.curl-multi-exec.php).

Another link that can be useful: http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/

Upvotes: 1

Related Questions