CODEWITHSUNDEEP
phpcurl
Revenant
Revenant

Reputation: 2984

PHP cURL executing a script doesn't work as expected

Function

function curlPost($url, $postArray = NULL, $timeout=2, $errorReport=FALSE) {
    # PREPARE THE POST STRING
    if ($postArray != NULL) {
        $postString = '';
        foreach ($postArray as $key => $val) {
            $postString .= urlencode($key) . '=' . urlencode($val) . '&';
        }
        $postString = rtrim($postString, '&');
    }

    # PREPARE THE CURL CALL
    $curl = curl_init();
    curl_setopt( $curl, CURLOPT_URL,            $url         );
    curl_setopt( $curl, CURLOPT_HEADER,         FALSE        );
    curl_setopt( $curl, CURLOPT_POST,           TRUE         );
    ($postArray != NULL) ? curl_setopt( $curl, CURLOPT_POSTFIELDS,     $postString ) : '';
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout     );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE         );

    # EXECUTE THE CURL CALL
    $htm = curl_exec($curl);
    $err = curl_errno($curl);
    $inf = curl_getinfo($curl);

    # ON FAILURE
    if (!$htm) {
        # PROCESS ERRORS HERE
        if ($errorReport) {
            echo "CURL FAIL: {$url} TIMEOUT={$timeout}, CURL_ERRNO={$err}";
            echo "<pre>\n";
            var_dump($inf);
            echo "</pre>\n";
            createLog("CURL FAIL: {$url} TIMEOUT={$timeout}, CURL_ERRNO={$err}");
        }
        curl_close($curl);
        return FALSE;
    }

    # ON SUCCESS
    curl_close($curl);
    return $htm;
 }

Problem

I'm using cURL in order to execute some bots. For example;

$url = 'http://siteurl.com/crons/transfer_members.php';
$response = curlPost($url, NULL, 10);
echo ($response) ? "{$url} executed" : "Couldn't execute {$url}";

It is working as expected but I have tons of members so I had to use pagination such as;

http://siteurl.com/crons/transfer_members.php?page=1

http://siteurl.com/crons/transfer_members.php?page=2 etc..

It is just transferring first result set and it is not going for other pages. If I execute it manually it is redirecting to next page (if there are records).

What should I do at this point? I'm quite stuck and I am out of ideas at the moment. I will be glad if anyone could help me out with this problem.

Note:

Everything is in the same domain. No tracing any referrer, not checking any cookie. Just pure PHP pages are called. If there is any pagination in the page which was called by cURL I use meta redirects to redirect to next page.

I don't want to use cURL inside of a loop. If called php page needs pagination I add following code inside of called php page;

# REDIRECT
echo '<meta http-equiv="refresh" content="' . $timer . ';url=' . $url . '">';

EXTRA INFORMATION

I have crons table in my database and I define $url with looping crons table's records. Some php pages might have pagination, some don't have any pagination. That's why I really would like to avoid using cURL calls inside of the loop to paginate.

If pagination is needed, there is pagination in called PHP page. If I visit the called page on browser than it is redirecting me to the next page. I expected the same thing when I call the page with cURL.

Meaning; if I visit http://siteurl.com/crons/transfer_members.php on my browser it is redirecting me to next page with meta redirect (example : http://siteurl.com/crons/transfer_members.php?page=x)

I expected the same thing with cURL. When cURL would call http://siteurl.com/crons/transfer_members.php it should go through next pages till all pages are finished. It is just visiting the first page and stopping there.

Upvotes: 0

Views: 1375

Answers (3)

Revenant
Revenant

Reputation: 2984

I just had a different approach. I returned http://siteurl.com/cron/transfer_members.php?page=n. If returned result starts with http:// or https:// then I called cURLonce again till returned data is finished.

I still didn't use cURL in the loop (it would cause other unwanted results) but still managed to make it work.

Special thanks to @Shiplu.

Upvotes: 0

Shiplu Mokaddim
Shiplu Mokaddim

Reputation: 57650

You need to simulate properly. There could be many reasons,

  1. siteurl.com might be tracing the referrer.
  2. You are not sending proper Request header also. Its just the curls default header that get passed.
  3. siteurl.com could be using Cookie. You are also not handling that too.

In fact it completely depends how siteurl.com facing your request.

Also you dont seem to use any pagination on $url = 'http://siteurl.com/crons/transfer_members.php'; It should be more like,

for($i=0;$i<100;$i++){
    $url = 'http://siteurl.com/crons/transfer_members.php?page='.$i;
    $response = curlPost($url, NULL, 10);
    echo ($response) ? "{$url} executed" : "Couldn't execute {$url}";
}

Upvotes: 1

Umbrella
Umbrella

Reputation: 4788

If you are directing the browser to successive pages by way of a script or meta tag, the browser is processing that for you, but cURL will not. One solution here is to do a quick str_pos for the code you use, and if found call cURL again. Do this in a loop and collect all the outputs until the string is not found, when you are done.

Upvotes: 0

Related Questions