CODEWITHSUNDEEP
phpmultithreadingcurldownload
Jacer Omri
Jacer Omri

Reputation: 1773

How can I download multiple parts of a file in parallel with PHP's curl library?

I decided to start a project about download acceleration with curl in PHP, using curl_multi functions.

Here is my code:

set_time_limit(0);
error_reporting(E_ALL);
$fileurl = "http://hq-scenes.com/tv.exe";
$filename = basename($fileurl);
$size = getFileSize($fileurl);
$splits = range(0, $size, round($size/5));
$megaconnect = curl_multi_init();
$partnames = array();
for ($i = 0; $i < sizeof($splits); $i++) {
    $ch[$i] = curl_init();
    curl_setopt($ch[$i], CURLOPT_URL, $fileurl);
    curl_setopt($ch[$i], CURLOPT_RETURNTRANSFER, 0); 
    curl_setopt($ch[$i], CURLOPT_FOLLOWLOCATION, 0); 
    curl_setopt($ch[$i], CURLOPT_VERBOSE, 1);
    curl_setopt($ch[$i], CURLOPT_BINARYTRANSFER, 1);
    curl_setopt($ch[$i], CURLOPT_FRESH_CONNECT, 0);
    curl_setopt($ch[$i], CURLOPT_CONNECTTIMEOUT, 10);
    $partnames[$i] = $filename . $i;
    $bh[$i] = fopen(getcwd(). '/' . $partnames[$i], 'w+');
    curl_setopt($ch[$i], CURLOPT_FILE, $bh[$i]);
        $x = ($i == 0 ? 0 : $splits[$i]+1);
        $y = ($i == sizeof($splits)-1 ? $size : $splits[$i+1]);
        $range = $x.'-'.$y;
    curl_setopt($ch[$i], CURLOPT_RANGE, $range);
    curl_setopt($ch[$i], CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.29 Safari/535.1");
    curl_multi_add_handle($megaconnect, $ch[$i]); 
}

$active = null;

do {
    $mrc = curl_multi_exec($megaconnect, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);

while ($active && $mrc == CURLM_OK) {
    if (curl_multi_select($megaconnect) != -1) {
        do {
            $mrc = curl_multi_exec($megaconnect, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
}
$final = fopen($filename, "w+");
for ($i = 0; $i < sizeof($splits); $i++) {
    $contents = fread($bh[$i], filesize($partnames[$i]));
    fclose($bh[$i]);
    fwrite($final, $contents);
}
fclose($final);

function getFileSize($url) {
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_NOBODY, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_HEADER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    $h = fopen('header', "w+");
    curl_setopt($ch, CURLOPT_WRITEHEADER, $h);

    $data = curl_exec($ch);
    curl_close($ch);

    if (preg_match('/Content-Length: (\d+)/', $data, $matches)) {
        return $contentLength = (int)$matches[1];
    }
    else return false;
}

Everything goes OK, except one thing:

The last part file doesn't reach the end of the file. the actual file size is : 3279848 bytes

ranges are:

0-655970
655971-1311940
1311941-1967910
1967911-2623880
2623881-3279848

part files with size

tv.exe0 655360
tv.exe1 655360
tv.exe2 655360
tv.exe3 655360
tv.exe4 655360

That makes the final file 3276800 bytes length, but it must be 3279848 bytes. And of course the executable didn't work :(

Notice that the part files have the same size. Even the last one, which should have some more bytes. So the problem is in the download range or something, not in the merge process.

What did I do wrong?

Upvotes: 3

Views: 4228

Answers (4)

Ardit Hyka
Ardit Hyka

Reputation: 784

I suggest you to add this after fclose($final); to delete the fileparts that are not needed anymore!

foreach($partnames as $files_to_delete){ 
    unlink($files_to_delete); 
}

Upvotes: 3

teemitzitrone
teemitzitrone

Reputation: 2260

you must set your filepointers to 0 before fread. reading xy bytes from end is 0 bytes ;) 

$final = fopen($filename, "w+");
for ($i = 0; $i < sizeof($splits); $i++) {
  fseek($bh[$i], 0, SEEK_SET);
  $contents = fread($bh[$i], filesize($partnames[$i]));
  fclose($bh[$i]);
  fwrite($final, $contents);
}

Upvotes: 2

Marc B
Marc B

Reputation: 360782

You'd want to use ceil() instead of round. Round may round DOWN, which'd chop off the end of the file. CEIL will round UP, guaranteeing that the specified range(s) covers the whole file:

$splits = range(0, $size, ceil($size/5));
                          ^^^^

e.g. If the file's size is 12, and you do round(13/5), you'll end up with 2.

Upvotes: 2

Jonathan Cremin
Jonathan Cremin

Reputation: 705

$size is not set to anything.

After setting it to the size you were expecting

655971 17 Aug 22:59 tv.exe0
655970 17 Aug 22:59 tv.exe1
655970 17 Aug 22:59 tv.exe2
655970 17 Aug 22:59 tv.exe3
655967 17 Aug 22:59 tv.exe4

Upvotes: 2

Related Questions