CODEWITHSUNDEEP
phpperformanceoptimizationfwritefile-put-contents
Ramon Bakker
Ramon Bakker

Reputation: 1085

PHP File Writing (fwrite / file_put_contents) speed/optimization

So, i have a database with big data. The data to use is currently about 2,6 GB.

All the data need to be written to a text file for later use in another scripts.

The data is being limited per file and splitted in multiple parts. 100 results per file (around 37MB each file). Thats about 71 files.

The data is json data that is being serialized and then encrypted with openssl.

The data is correctly being written to the files, untill the max execution time is reached after 240 seconds. That's after about 20 files...

Well, i can just extend that time, but thats not the problem.

The problem is the following:

Writing file 1-6:   +/- 5 seconds
Writing file 7-8:   +/- 7 seconds
Writing file 9-11:  +/- 12 seconds
Writing file 12-14: +/- 17 seconds
Writing file 14-16: +/- 20 seconds
Writing file 16-18: +/- 23 seconds
Writing file 19-20: +/- 27 seconds

Note: time is needed time per file

In other words, with every file im writing, the writing time per file goes significantly up, what causes the script to be slow offcourse.

The structure of the script is a bit like this:

$needed_files = count needed files/parts

for ($part=1; $part<=$needed_files; $part++) { // Loop throught parts
    $query > mysqli select data
    $data > json_encode > serialize > openssl_encrypyt
    file_put_contents($filename.$part, $data, LOCK_EX);
}

WORKING CODE AFTER HELP

$notchDetails = mysqli_query($conn, "SELECT * FROM notches WHERE projectid = ".$projectid."");

$rec_count = 0;
$limit = 100;
$part = 1;

while ($notch = mysqli_fetch_assoc($notchDetails)) {

    $data1[] = $notch;
    $rec_count++;

    if ($rec_count >= $limit) {

        $data = json_encode($data1);
        $data = openssl_encrypt(bin2hex($data), "aes128", $pass, false, $iv);
        $filename = $mainfolder."/".$projectfolder."/".$subfolder."/".$fname.".part".$part."".$fext;
        file_put_contents($filename, $data, LOCK_EX);

        $part++;
        $rec_count = 0;
        $data = $data1 = "";

    }

}
if ($data1 != "") {
    $data = json_encode($data1);
    $data = openssl_encrypt(bin2hex($data), "aes128", $pass, false, $iv);
    $filename = $mainfolder."/".$projectfolder."/".$subfolder."/".$fname.".part".$part."".$fext;
    file_put_contents($filename, $data, LOCK_EX);
}

mysqli_free_result($notchDetails);

Upvotes: 0

Views: 3535

Answers (1)

RiggsFolly
RiggsFolly

Reputation: 94662

Personally I would have coded this as a single SELECT with no LIMIT and then based on a $rec_per_file = ?; write the outputs from within the single while get results loop

Excuse the cryptic code, you didnt give us much of a clue

<?php
//ini_set('max_execution_time', 600);    // only use if you have to

$filename = 'something';
$filename_suffix = 1;

$rec_per_file = 100;

$sql = "SELECT ....";

Run query

$rec_count = 0;

while ( $row = fetch a row ) {

    $data[] = serialize > openssl_encrypyt

    $rec_count++;

    if ( $rec_count >= $rec_per_file ) {

        $json_string = json_encode($data);

        file_put_contents($filename.$filename_suffix, 
                          $json_string, 
                          LOCK_EX);

        $filename_suffix++; // inc the suffix
        $rec_count = 0;     // reset counter
        $data = array();    // clear data

        // add 30 seconds to the remaining max_execution_time
        // or at least a number >= to the time you expect this
        // while loop to get back to this if statement
        set_time_limit(30);
    }
}
// catch the last few rows
$json_string = json_encode($data);
file_put_contents($filename.$filename_suffix, $data, LOCK_EX);

Also I am not sure why you would want to serialize() and json_encode()

I had a thought, based on your comment about execution time. If you place a set_time_limit(seconds) inside the if inside the while loop it might be cleaner, and you would not have to set ini_set('max_execution_time', 600); to a very large number, which if you have a real error in here may cause PHP continue processing for a long time before kicking the script out.

From the manual:

Set the number of seconds a script is allowed to run. If this is reached, the script returns a fatal error. The default limit is 30 seconds or, if it exists, the max_execution_time value defined in the php.ini.

When called, set_time_limit() restarts the timeout counter from zero. In other words, if the timeout is the default 30 seconds, and 25 seconds into script execution a call such as set_time_limit(20) is made, the script will run for a total of 45 seconds before timing out.

Upvotes: 1

Related Questions