CODEWITHSUNDEEP
phpfilesplit
Mihir
Mihir

Reputation: 1176

Split big files using PHP

I want to split huge files (to be specific, tar.gz files) in multiple part from php code. Main reason to do this is, php's 2gb limit on 32bit system.

SO I want to split big files in multiple part and process each part seperately.

Is this possible? If yes, how?

Upvotes: 13

Views: 27829

Answers (9)

Guest
Guest

Reputation: 1

Splits are named as filename.part0 filename.part1 ...

<?php
function fsplit($file,$buffer=1024){
    //open file to read
    $file_handle = fopen($file,'r');
    //get file size
    $file_size = filesize($file);
    //no of parts to split
    $parts = $file_size / $buffer;

    //store all the file names
    $file_parts = array();

    //path to write the final files
    $store_path = "splits/";

    //name of input file
    $file_name = basename($file);

    for($i=0;$i<$parts;$i++){
        //read buffer sized amount from file
        $file_part = fread($file_handle, $buffer);
        //the filename of the part
        $file_part_path = $store_path.$file_name.".part$i";
        //open the new file [create it] to write
        $file_new = fopen($file_part_path,'w+');
        //write the part of file
        fwrite($file_new, $file_part);
        //add the name of the file to part list [optional]
        array_push($file_parts, $file_part_path);
        //close the part file handle
        fclose($file_new);
    }    
    //close the main file handle

    fclose($file_handle);
    return $file_parts;
}
?>

Upvotes: 0

RobertPitt
RobertPitt

Reputation: 57268

A simple method (if using Linux based server) is to use the exec command and to run the split command:

exec('split Large.tar.gz -b 4096k SmallParts'); // 4MB parts
/*    |     |            |      | |
      |     |            |______| |
      App   |                 |   |_____________
            The source file   |                 |
                              The split size    Out Filename
*/

See here for more details: http://www.computerhope.com/unix/usplit.htm

Or you can use: http://www.computerhope.com/unix/ucsplit.htm

exec('csplit -k -s -f part_ -n 3 LargeFile.tar.gz');

PHP runs within a single thread and the only way to increase this thread count is to create child process using the fork commands.

This is not resource friendly. What I would suggest is to look into a language that can do this fast and effectively. I would suggest using node.js.

Just install node on the server and then create a small script, called node_split for instance, that can do the job on its own for you.

But I do strongly advise that you do not use PHP for this job but use exec to allow the host operating system to do this.

Upvotes: 10

user4931222
user4931222

Reputation: 

$handle = fopen('source/file/path','r'); 
        $f = 1; //new file number
        while(!feof($handle))
        {
            $newfile = fopen('newfile/path/'.$f.'.txt','w'); //create new file to write to with file number
            for($i = 1; $i <= 5000; $i++) //for 5000 lines
            {
                $import = fgets($handle);
                //print_r($import);
                fwrite($newfile,$import);
                if(feof($handle))
                {break;} //If file ends, break loop
            }
            fclose($newfile);
            $f++; //Increment newfile number
        }
        fclose($handle);

Upvotes: 1

Bhupendra Kanojiya
Bhupendra Kanojiya

Reputation: 11

function split_file($source, $targetpath='/split/', $lines=1000){

    $i=0;
    $j=1;
    $date = date("m-d-y");
    $buffer='';

    $handle = fopen ($_SERVER['DOCUMENT_ROOT'].$source, "r");

    while (!feof ($handle)) {
        $buffer .= fgets($handle, 4096);
        $i++;
        if ($i >= $lines) {
            $fname = $_SERVER['DOCUMENT_ROOT'].$targetpath."part_".$date.$j.".txt";

                 $fhandle = fopen($fname, "w") or die($php_errormsg);

            if (!$fhandle) {
                echo "Cannot open file ($fname)";
                //exit;
            }


            if (!fwrite($fhandle, $buffer)) {
                echo "Cannot write to file ($fname)";
                //exit;
            }
            fclose($fhandle);
            $j++;
            $buffer='';
            $i=0;
            $line+=10; // add 10 to $lines after each iteration. Modify this line as required
        }
    }
    fclose ($handle);
}

Upvotes: 1

alex
alex

Reputation: 490173

My comment was voted up twice, so maybe my guess was onto something :P

If on a unix environment, try this...

exec('split -d -b 2048m file.tar.gz pieces');

split

Your pieces should be pieces1, pieces2, etc.

You could get the number of resulting pieces easily by using stat() in PHP to get the file size and then do the simple math (int) ($stat['size'] / 2048*1024*1024) (I think).

Upvotes: 12

Vamsi Krishna B
Vamsi Krishna B

Reputation: 11490

HJSPLIT

http://www.hjsplit.org/php/

Upvotes: 3

Johann du Toit
Johann du Toit

Reputation: 2667

This would probably be possible in php, but php was built for web development and trying to this whole operation in one request will result in the request timing out.

You could however use another language like java or c# and build a background process that you can notify from php to perform the operation. Or even run from php, depending on your Security settings on the host.

Upvotes: 0

HTDutchy
HTDutchy

Reputation: 1130

PHP itself might not be able to... If you can figure out how to do this from your computers' command line, You should be able to then execute these commands using exec();

Upvotes: 1

Vincent Mimoun-Prat
Vincent Mimoun-Prat

Reputation: 28541

  • If you want to split files which are already on server, you can do it (simply use the file functions fread, fopen, fwrite, fseek to read/write part of the file).
  • If you want to split files which are uploaded from the client, I am afraid you cannot.

Upvotes: 0

Related Questions