CODEWITHSUNDEEP
filesystemscluster-computing
RandomITGuy
RandomITGuy

Reputation: 35

Creating a Cluster Fileserver System

I Currently Have 3 Fileserver each has a raid 6 array of 24 disks.

The Question is this is there any way to make them work as one big drive rather that 3 seperate systems. I need more throughput and i was thinking this was a possibilty. Maybe a Distrubted Filesystem like Hadoop?

Upvotes: 2

Views: 283

Answers (1)

David Gruzman
David Gruzman

Reputation: 8088

The answer depends on the intended usage of the data on this hardware. Hadoop file system HDFS - is something suited for very special need of the Map-Reduce processing. Main limitations, which are ok for its intended use, but problematic for others are:
a) Files can not be edited, but only appended.
b) There will be a problem to stoe many small files. It is designed for file of size 64 MB and more. The cause of this limitation that all metadata is stored in memory.
c) It is not posix compliant FS, so you can not mount it and use as regular file system by the application unaware of HDFS.

I would consider options like GlusterFS, Ceph or Lustre which are built for the cases similar to one you describe. More information is needed to give good advice of selecting one of them.

Upvotes: 3

Related Questions