CODEWITHSUNDEEP
phpfile
Alma Do
Alma Do

Reputation: 37365

Write data to start of file when not enough space

The problem

How to write data to start of file if I have not enough space to allocate it in RAM and I have not enough space to make it's copy on current FS partition? I.e. I have a file with 100Mb size, I have 30Mb memory limit in my PHP script (and it can not be adjusted in any way) and I have only 50Mb free on my current FS partition. I want to add 2-10 rows to file (it's definitely less than remaining 50Mb FS space)

Some background

I know about XY-problem and agree that it's true for this case. But to reconsider this case I'll need to change significant part of current application (actually, it went from previous team) and, may be, API of other applications that using this file.

My attempt

I have not found solution for this yet. My previous approach was - to use some network buffer (i.e. to connect to some external storage, such as MySQL, for example - it's located on another machine where there is enough space to write file's copy)

The question

So, is it possible to write data to file's start when I have not enough space to allocate it in RAM and have not enough space to create file's copy on FS? Is using network (external) storage the only solution?

Upvotes: 2

Views: 769

Answers (3)

Kirow
Kirow

Reputation: 1210

I've tried code suggested by @AlmaDo, don't try it on real projects, or you will be burn in hell, it is VERY slow. (60MB file - processing 19minutes)

You can run shell script - https://stackoverflow.com/a/9533736/2064576 (processed 420ms, can not understand how much memory does it use) Or try this php script - https://stackoverflow.com/a/16813550/2064576 (160ms, worked with memory_limit=3M, not worked with 2M)

Upvotes: 0

Alma Do
Alma Do

Reputation: 37365

@deceze hint me great idea. So I've finished with:

function reverseFile($sIn, $sOut, $bRemoveSource=false)
{
        $rFile = @fopen($sIn, 'a+');
        $rTemp = @fopen($sOut,'a+');
        if(!$rFile || !$rTemp)
        {
                return false;
        }
        $iPos = filesize($sIn)-1;
        while($iPos>=0)
        {
                fseek($rFile, $iPos, SEEK_SET);
                fwrite($rTemp, $tmp=fread($rFile, 1));
                ftruncate($rFile, $iPos>0?$iPos:0);
                clearstatcache();
                $iPos--;
        }
        fclose($rFile);
        fclose($rTemp);
        if($bRemoveSource)
        {
                unlink($sIn);
        }
        return true;
}

function writeReverse($sFile, $sData, $sTemp=null)
{
        if(!isset($sTemp))
        {
                $sTemp=$sFile.'.rev';
        }
        if(reverseFile($sFile, $sTemp, 1))
        {
                file_put_contents($sTemp, strrev($sData), FILE_APPEND);
                return reverseFile($sTemp, $sFile, 1);
        }
        return false;
}

-it will be quite slow, but recoverable if process is interrupted (simply look to .rev file)

Thanks to all who participated in this.

Upvotes: 1

deceze
deceze

Reputation: 522016

Say you want to write 2K to the beginning of a file, your only real option is to:

  • open the file
  • read as much from the end of the file as you can fit into memory
  • write it back into the file 2K later than you started to read
  • continue with the previous block of data until you have shifted the entire content of the file 2K towards the end
  • write your 2K to the beginning

To visualize that:

|------------------------|

|-----------------XXXXXXX|
                  ------>

|-------------------XXXXXXX|

|----------XXXXXXX---------|
           ------>

|------------XXXXXXX-------|

...repeat...

Note that this is a very unsafe operation which edits the file in place. If the process crashes, you're left with a file in an inconsistent state. If you don't have enough room on disk to duplicate a file you arguably shouldn't work with that file and expand your storage capacity first.

Upvotes: 2

Related Questions