CODEWITHSUNDEEP
phpreadfile
alsanal
alsanal

Reputation: 183

read big file in php (more than 500mb)

I need to read a large file to find some labels and create a dynamic form. I can not use file() or file_get_contents() because the file size. If I read the file line by line with the following code

set_time_limit(0);    
$handle = fopen($file, 'r');
    set_time_limit(0);
    if ($handle) {
        while (!feof($handle)) {
            $line = fgets($handle);
            if ($line) {            
             //do something.
            }
        }
    }

    echo 'Read complete';

I get the following error in Chrome:

Error 101 (net::ERR_CONNECTION_RESET)

This error occurs after several minutes so that the constant max_input_time, I think not is the problem.(is set to 60).

Upvotes: 0

Views: 1615

Answers (3)

Or Cohen
Or Cohen

Reputation: 1546

Generally long running processes should not be done while the users waits for them to complete. I'd recommend using a background job oriented tool that can handle this type of work and can be queried about the status of the job (running/finished/error).

My first guess is that something in the middle breaks the connection because of a timeout. Whether it's a timeout in the web server (which PHP cannot know about) or some firewall, it doesn't really matter, PHP gets a signal to close the connection and the script stops running. You could circumvent this behaviour by using ignore-user-abort(true), this along with set_time_limit(0) should do the trick.

The caveat is that whatever caused the connection abort will still do it, though the script would still finish it's job. One very annoying side effect is that this script could possibly be executed multiple times in parallel without neither of them ever completing.

Again, I recommend using some background task to do it and an interface for the end-user (browser) to verify the status of that task. You could also implement a basic one yourself via cron jobs and database/text files that hold the status.

Upvotes: 0

Navneet Singh
Navneet Singh

Reputation: 1198

Set the memory limit ini_set("memory_limit","600M");also you need to set the time out limit 

set_time_limit(0);

Upvotes: 1

Bastiaan Zwanenburg
Bastiaan Zwanenburg

Reputation: 143

What browser software do you use? Apache, nginx? You should set the max accepted file upload at somewhere higher than 500MB. Furthermore, the max upload size in the php.ini should be bigger than 500MB, too, and I think that PHP must be allowed to spawn processes larger than 500MB. (check this in your php config).

Upvotes: 1

Related Questions