CODEWITHSUNDEEP
linuxcaching
Eugene Ryabtsev
Eugene Ryabtsev

Reputation: 2301

Telling Linux not to keep a file in the cache when it is written to disk

I am writing a large file to disk from a user-mode application. In parallel to it, I am writing one or more smaller files. The large file won't be read back anytime soon, but the small files could be. I have enough RAM for the application + smaller files, but not enough for the large file. Can I tell the OS not to keep parts of the large file in cache after they are written to disk so that more cache is available for smaller files? I still want writes to the large file be fast enough.

Upvotes: 0

Views: 686

Answers (1)

Basile Starynkevitch
Basile Starynkevitch

Reputation: 1

Can I tell the OS not to keep parts of the large file in cache ?

Yes, you probably want to use some system call like posix_fadvise(2) or madvise(2). In weird cases, you might use readahead(2) or userfaultfd(2) or Linux-specific flags to mmap(2). Or very cleverly handle SIGSEGV (see signal(7), signal-safety(7) and eventfd(2) and signalfd(2)) You'll need to write your C program doing that.

But I am not sure that it is worth your development efforts. In many cases, the behavior of a recent Linux kernel is good enough.

See also proc(5) and linuxatemyram.com

You many want to read the GC handbook. It is relevant to your concerns

Conbsider studying for inspiration the source code of existing open-source software such as GCC, Qt, RefPerSys, PostGreSQL, GNU Bash, etc...

Most of the time, it is simply not worth the effort to explicitly code something to manage your page cache.

I guess that mount(2) options in your /etc/fstab file (see fstab(5)...) are in practice more important. Or changing or tuning your file system (e.g. ext4(5), xfs(5)..). Or read(2)-ing in large pieces (1Mbytes).

Play with dd(1) to measure. See also time(7)

Most applications are not disk-bound, and for those who are disk bound, renting more disk space is cheaper that adding and debugging extra code.

don't forget to benchmark, e.g. using strace(1) and time(1)

PS. Don't forget your developer costs. They often are a lot above the price of a RAM module (or of some faster SSD disk).

Upvotes: 1

Related Questions