CODEWITHSUNDEEP
fileunixram
Maverickgugu
Maverickgugu

Reputation: 777

How to read a large file in UNIX/LINUX with limited RAM?

I have a large text file to be opened (eg- 5GB size). But with a limited RAM (take 1 GB), How can I open and read the file with out any memory error? I am running on a linux terminal with with the basic packages installed.

This was an interview question, hence please do not look into the practicality.

I do not know whether to look at it in System level or programmatic level... It would be great if someone can throw some light into this issue.

Thanks.

Upvotes: 3

Views: 2912

Answers (3)

badawi
badawi

Reputation: 895

[just comment] If you are going to use system calls (open() and read()), then reading character by character will generate a lot of system calls that severely slow down your application. Even with the existence of the buffer cache (or disk file), system calls are expensive.

It is better to read block by block where block size "SHOULD" be more than 1MB. In case of 1MB block size, you will issue 5*1024 system calls.

Upvotes: 2

halm
halm

Reputation: 377

What do they want you to do with the file? Are you looking for something? Extracting something? Sorting? This will affect your approach.

It may be sufficient to read the file line by line or character by character if you're looking for something. If you need to jump around the file or analyze sections of it, then most likely want to memory map it. Look up mmap(). Here's an short article on the subject:memory mapped i/o

Upvotes: 2

Ry-
Ry-

Reputation: 224903

Read it character by character... or X bytes by X bytes... it really depends what you want to do with it... As long as you don't need the whole file at once, that works.

(Ellipses are awesome)

Upvotes: 5

Related Questions