CODEWITHSUNDEEP
amazon-web-servicesscalabilitydistributedhigh-availability
Anatoly
Anatoly

Reputation: 15530

Cloud hosting - shared storage with direct access

We have an application deployed across AWS with using EC2, EBS services. The infrastructure dropped by layers (independent instances):

Application and Database instance use number of EBS block storage devices (root, data), which help us to attach/detach them and do EBS snapshots to S3. It's pretty default way how AWS works.

But EBS should be located in a specific zone and can be attached to one instance only in the same time.

Media server is one of bottlenecks, so we'd like to scale them with master/slave schema. So for the media server storage we'd like to try distributed file systems can be attached to multiple servers. What do you advice?

Upvotes: -1

Views: 576

Answers (2)

user297080
user297080

Reputation:

If you're not Facebook or Amazon, then you have no real reason to use something as elaborate as Hadoop or Cassandra. When you reach that level of growth, you'll be able to afford engineers who can choose/design the perfect solution to your problems.

In the meantime, I would strongly recommend GlusterFS for distributed storage. It's extremely easy to install, configure and get up and running. Also, if you're currently streaming files from local storage, you'll appreciate that GlusterFS also acts as local storage while remaining accessible by multiple servers. In other words, no changes to your application are required.

I can't tell you the exact configuration options for your specific application, but there are many available such as distributed, replicated, striped data. You can also play with cache settings to avoid hitting disks on every request, etc.

One thing to note, since GlusterFS is a layer above the other storage layers (particularly with Amazon), you might not get impressive disk performance. Actually it might be much worst than what you have now, for the sake of scalability... basically you could be better-off designing your application to serve streaming media from a CDN who already has the correct infrastructure for your type of application. It's something to think about.

Upvotes: 1

Poni
Poni

Reputation: 11327

HBase/Hadoop

Cassandra

MogileFS

Good same question (if I understand correctly):
Lustre, Gluster or MogileFS?? for video storage, encoding and streaming

There are many distributed file systems, just find the one you need.
The above are just part which I personally know (haven't tested them).

Upvotes: 0

Related Questions