CODEWITHSUNDEEP
linuxfilesystems
Devrim
Devrim

Reputation: 2914

FS based on a database without using fuse

To serve millions of files out of a single directory, being able to connect to a drive from hundreds of endpoints, and for some other reasons (to avoid gluster/nfs/all fs based networking solutions), I want to evaluate the possibility of making a filesystem that's based on a mongodb (or any other).

Basically, it works like fusefs, every single file is kept in mongo gridfs. In theory, I do,

mount mongodbfs /mountPoint mongodb://localhost

then when i say touch /mountPoint/test.txt this file is inserted into mongodb. This FS will also store uid/gid and perms with the file, we can throw hundreds of servers to it, and no useradd will be necessary. I'm not thinking to include all the features of FS, just the ones we need.

My question is, how do I start my quest in finding resources, books, links, people, developers who'd help me implement this? at least a proof of concept. Is it feasible? What should I expect as a timeline for such undertaking?

Please only think about gazillion small files and folders.

ps: after a few days of research i think this is the direction i'm heading http://www.ibm.com/developerworks/library/l-sc12.html http://www.flipcode.com/archives/Programming_a_Virtual_File_System-Part_I.shtml

ps2: i'm aware of the difficulty of this undertaking. however we're willing to set aside a serious budget and willing to form a serious team implementing it - only after we make sure that this isn't a black hole (thus the question).

Upvotes: 4

Views: 5030

Answers (1)

voretaq7
voretaq7

Reputation: 212

Your most frequent piece of advice here is going to be "Use FUSE". This is excellent advice, and you would do well to heed it (As Sciurus pointed out there's already gridfs-fuse which is pretty close to what you want).

That said, if you want to take the long, hard road of pain and suffering (writing your own filesystem), you almost certainly want to take an operating systems course at a local university, or look at some online course materials ("Write a simple FS" is usually a small project. The filesystems typically suck because they're academic toys).
Follow that up with Linux File Systems (Moshe Bar) and a thorough reading of some simple filesystem drivers to see the basic skeleton of what you'll need to do.

As far as timeline, if you're a decent coder you can write a basic filesystem in a few days to a week (but it will SUCK). I wouldn't even guess how long it would take to write a GOOD filesystem -- UFS/FFS (the BSD filesystem) has been under continuous development since at least the late 1970s/early 1980s, and improvements/enhancements/bug fixes still pop up occasionally. Sun/Oracle's ZFS has gone through over 20 iterations in its relative short (6-year) life, though admittedly much of that is related to volume management capabilities.

Upvotes: 7

Related Questions