CODEWITHSUNDEEP
c++linuxboostmmap
jb.
jb.

Reputation: 23995

How can detect error when boost memory mapped file allocated more disk space than is free on HDD

In my modelling code I use boost memory mapped files, to allocate large-ish arrays on disk.

It works well, but I couldn't find a way to detect situation in which I allocate array which is larger than free space on disk drivw. For example following code will execute happily (assuming that I have less than 8E9 bytes of free space on HDD):

    boost::iostreams::mapped_file_params file_params;

    file_params.path = path;
    file_params.mode = std::ios::in | std::ios::out;
    file_params.new_file_size = static_cast<size_t>(8E9); # About 10GB
    file_params.length = static_cast<size_t>(8E9);

    boost::iostreams::mapped_file result;

    result.open(file_params);

I can even work on resuld.data() until I write to part of memory which is not allocted (becaue of missing space on HDD) and then I get a following error: 

 memory access violation at address: 0x7e9e2cd1e000: non-existent physical address

Is there any way to detect this error before I get cryptic memory access violation?

I actually tested this: if file is bigger than avilable free space on partition it code has memory access violation, if it is smaller the code works (I tested it by changing space free on the partition not by editing the code).

Possible solutions

If I std:fill file contents with zeroes I still get memory access violation, but this error is located near the allocation and easier to debug. I'd rather want some way to raise an exception.

Upvotes: 1

Views: 590

Answers (2)

sehe
sehe

Reputation: 393694

You can use fallocate or posix_fallocate to actually reserve space for the file up front. This way you know you won't ever "over-commit". It has a performance drawback, of course upon initial creation.

For security reasons the OS is likely to zero out the blocks on fallocate.

fallocate lets you do unwritten extents, but it still zeros upon first access. On windows, this can be combated using, SetFileValidData lets you bypass even that.

Note that Linux with O_DIRECT + fallocate(), still uses considerable CPU (as opposed to Windows' SetFileValidData), and although IO bandwidth is usually the bottleneck, this could still have noticeable performance effect if you are doing much CPU work at the same time.

Upvotes: 1

Turbo J
Turbo J

Reputation: 7701

Is there any way to detect this error before I get cryptic memory access violation?

When you just change the size of a file it will be sparse, that means the areas without data do not consume disk space. Space is allocated during writes - and can create out of diskspace errors.

A way to solve the problem would be to write (dummy) data to the file instead of just changing the size. That takes more time but you would get the out of diskspace during this first write cycle only, as the file has its final size afterwards.

Upvotes: 1

Related Questions