CODEWITHSUNDEEP
clinuxmemory-mapping
user2882307
user2882307

Reputation:

memory-mapping pre-defined areas of drive

I want to read very large chunks of data using memory mapped io.

These large chunks of data are comming from a harddisk, no file system just data.

Now before I start this whole ordeal I want to know 2 things.

  1. is it possible to memory map only specific parts into memory after eachother and then read is sequentally? First instance I have a harddrive where I want to read 10 chunks of 100mb but each chunk is separated by 1gb of data. is it possible to memory map those 10 chunks of 100mb one after the other so I can acces it like if they were one after the other?

  2. Can I memory map huge amount of data? e.g let`s say I have a 10tb disk. is it possible to memory the entire disk? I use a 64bit OS.

I hope someone can clarify!

Upvotes: 2

Views: 506

Answers (1)

fuz
fuz

Reputation: 93044

On Linux, you can use the mmap() system call to map files (even block devices) into memory. If you don't know how mmap() works, consult the man page before continuing with this answer.

The mmap() call allows you to specify a base address for the mapping you want to create. POSIX specifies that the operating system may take this base address as a hint on where to place the mapping. On Linux, mmap() will place the mapping on the address you request if it is a page boundary (i.e. dividable by 4096). You can specify MAP_FIXED to make sure that the mapping is placed where you want it, but the kernel might tell you that this is not possible.

You can try to map the chunks you want one-after-another using the approach above but this obviously will only work if your chunks have sizes that are multiples of the page size (i.e. 4096 bytes). I would not advise you to do this as it might break on a different page size / configuration.

Mapping the entire disk should be possible depending on your memory configuration. You might need to configure the overcommiting behavior of your system for this.

I suggest you to try out if mapping the entire disk works.

Upvotes: 1

Related Questions