CODEWITHSUNDEEP
linuxshared-memorymmaptlbinode
Nathan Kurz
Nathan Kurz

Reputation: 1699

Inserting pages into large mmap() files without copying data

I'm wondering if there is a way to insert blank pages near the beginning of a large (multi-GB) file that I have open with mmap(). Obviously it would be possible to add a page or two to the end, and move everything forward with memcpy(), but this would dirty every page and require an awful long time when eventually flushed to disk.

I'm guessing that a solution would require some complex coordination between a customized filesystem and manual manipulation of the page tables: add a block to the inode, somehow update the cached pages in the VMM to reflect this, then somehow swizzle the page table to match. This sounds non-trivial, which makes me wonder if there's a better way.

This is intended as a somewhat deep question about memory and file manipulation on Linux, although I'd be happy to hear about how this can be done in other systems. I'm not particularly interested in workarounds that involve making the copying more efficient, although a technique that requires remapping but avoids the disk IO would be a good start.

Upvotes: 0

Views: 581

Answers (1)

johnnycrash
johnnycrash

Reputation: 5344

Embed a simple FAT in your file. For instance, the first 4k of the file would be a the FAT page. Data would be in following pages. The first FAT page could link to other FAT pages as your file grew. Each entry in the fat would be a data page index and the index of the next FAT entry. A FAT entry would be the page of the FAT and the index on that page of the entry itself. I think you get the idea. The FAT entries are A linked list. The FAT pages are a linked list. The FAT entries link data pages. This should be enough information to use remap_file_pages() to make your file look contiguous in memory even though its not contiguous on the disk.

Upvotes: 2

Related Questions