CODEWITHSUNDEEP
phpfile
user4951
user4951

Reputation: 33090

How to ocassionally shorten files?

This is the code I use:

if(mt_rand(0,20000)==0)
{
    $lines = file($fileName);
    if (count($lines)>50000)
    {
        $lines=array_slice($lines, -50000, 50000, true);
    }
    $result=implode("\n",lines);
    file_put_contents($fileName, $result . "\n",FILE_APPEND);
}

I often got this error:

[25-Nov-2013 23:20:40 UTC] PHP Fatal error:  Allowed memory size of 33554432 bytes exhausted (tried to allocate 33 bytes) in /home/datetrng/public_html/checkblocked.php on line 40
[26-Nov-2013 02:41:54 UTC] PHP Fatal error:  Allowed memory size of 33554432 bytes exhausted (tried to allocate 27 bytes) in /home/datetrng/public_html/checkblocked.php on line 40
[26-Nov-2013 09:56:49 UTC] PHP Fatal error:  Allowed memory size of 33554432 bytes exhausted (tried to allocate 72 bytes) in /home/datetrng/public_html/checkblocked.php on line 40
[26-Nov-2013 12:44:32 UTC] PHP Fatal error:  Allowed memory size of 33554432 bytes exhausted (tried to allocate 2097152 bytes) in /home/datetrng/public_html/checkblocked.php on line 40
[26-Nov-2013 13:53:31 UTC] PHP Fatal error:  Allowed memory size of 33554432 bytes exhausted (tried to allocate 2097152 bytes) in /home/datetrng/public_html/checkblocked.php on line 40

I guess reading the whole file may not be a good idea if we just want to shorten a file by erasing the beginning of the file.

Knows any alternative?

Upvotes: 0

Views: 75

Answers (3)

srain
srain

Reputation: 9082

I think you only want the last 50000 lines in your file.

if(mt_rand(0,20000)==0)
{
    $tmp_file = $fileName . '.tmp';
    $cmd = "tail -n 50000  $fileName > $tmp_file";
    exec($cmd);
    rename($tmp_file, $fileName);
}

update for pure php

I make a file about 100,000 lines:

<?php
$file_name = 'tmp.dat';
$f = fopen($file_name, 'w');
for ($i = 0; $i < 1000000; $i++)
{
    fwrite($f, str_pad($i, 100, 'x') . "\n");
}
fclose($f);

This file is about 97M.

[huqiu@localhost home]$ ll -h  tmp.dat
-rw-rw-r-- 1 huqiu huqiu 97M Nov 27 06:08 tmp.dat

read the last 50000 lines

<?php
$file_name = 'tmp.dat';
$remain_count = 50000;

$begin_time = microtime(true);
$temp_file_name = $file_name . '.tmp';

$fp = fopen($file_name, 'r');
$total_count = 0;
while(fgets($fp))
{
    $total_count++;
}
echo 'total count: ' . $total_count . "\n";
if ($total_count > $remain_count)
{
    $start = $total_count - $remain_count;
    echo 'start: ' . $start . "\n";
    $temp_fp = fopen($temp_file_name, 'w');

    $index = 0;
    rewind($fp);
    while($line = fgets($fp))
    {
        $index++;
        if ($index > $start)
        {
            fwrite($temp_fp, $line);
        }
    }
    fclose($temp_fp);
}
fclose($fp);
echo 'time: ' . (microtime(true) - $begin_time), "\n";

rename($temp_file_name, $file_name);

elapsed time: 0.63908791542053

total count: 1000000
start: 950000
time: 0.63908791542053

the result:

[huqiu@localhost home]$ ll -h tmp.dat  
-rw-rw-r-- 1 huqiu huqiu 4.9M Nov 27 06:23 tmp.dat

Upvotes: 1

DevlshOne
DevlshOne

Reputation: 8457

Why not fseek the pointer to a position past the point you want to eliminate? You might also have better luck using fpassthru to save some memory.

Upvotes: 0

flcoder
flcoder

Reputation: 710

fopen fwrite fseek will probably come in handy for you

Upvotes: 2

Related Questions