CODEWITHSUNDEEP
databaseperllarge-filesreadlines
user1645240
user1645240

Reputation: 55

Reading a particular line by line number in a very large file

The file will not fit into memory. It is over 100GB and I want to access specific lines by line number. I do not want to count line by line until I reach it.

I have read http://docstore.mik.ua/orelly/perl/cookbook/ch08_09.htm

When I built an index using the following methods, the line return works up to a certain point. Once the line number is very large, the line being returned is the same. When I go to the specific line in the file the same line is returned. It seems to work for line numbers 1 through 350000 (approximately);

 # usage: build_index(*DATA_HANDLE, *INDEX_HANDLE) 
    sub build_index {
        my $data_file  = shift;
        my $index_file = shift;
        my $offset     = 0;

        while (<$data_file>) {
            print $index_file pack("N", $offset);
            $offset = tell($data_file);
        }
    }

    # usage: line_with_index(*DATA_HANDLE, *INDEX_HANDLE, $LINE_NUMBER)
    # returns line or undef if LINE_NUMBER was out of range
    sub line_with_index {
        my $data_file   = shift;
        my $index_file  = shift;
        my $line_number = shift;

        my $size;               # size of an index entry
        my $i_offset;           # offset into the index of the entry
        my $entry;              # index entry
        my $d_offset;           # offset into the data file

        $size = length(pack("N", 0));
        $i_offset = $size * ($line_number-1);
        seek($index_file, $i_offset, 0) or return;
        read($index_file, $entry, $size);
        $d_offset = unpack("N", $entry);
        seek($data_file, $d_offset, 0);
        return scalar(<$data_file>);
    }

I've also tried using the DB_file method, but it seems to take a very long time to do the tie. I also don't really understand what it means for "DB_RECNO access method ties an array to a file, one line per array element." Tie does not read the file into the array correct?

Upvotes: 0

Views: 387

Answers (1)

ikegami
ikegami

Reputation: 385789

pack N creates a 32-bit integer. The maximum 32-bit integer is 4GB, so using that to store indexes into a file that's 100GB in size won't work.

Some builds use 64-bit integers. On those, you could use j.

Some builds use 32-bit integers. tell returns a floating-point number on those, allowing you to index files up to 8,388,608 GB in size losslessly. On those, you should use F.

Portable code would look as follows:

use Config qw( %Config );
my $off_t = $Config{lseeksize} > $Config{ivsize} ? 'F' : 'j';

...
print $index_file pack($off_t, $offset);
...

Note: I'm assuming the index file is only used by the same Perl that built it (or at least one with with the same integer size, seek size and machine endianness). Let me know if that assumption doesn't hold for you.

Upvotes: 4

Related Questions