CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-ec2
Jordan
Jordan

Reputation: 1614

AWS services appropriate for concurrent access to a resource

I'm designing a system where a cluster of EC2 instances do some computing and then update a large file continually. What would be ideal is if I could have the file in S3, and have all the instances take turns writing to it one at a time, performing calculations while they wait. As it stands if 2 instances PUT to S3 at the same time, 1 will simply override the other.

How can I solve this concurrency issue?

Upvotes: 1

Views: 548

Answers (2)

Akira
Akira

Reputation: 4071

AWS has a preview service called EFS (http://aws.amazon.com/documentation/efs/) that is an NFS4 that can be shared among EC2 instances. But such service alone does not solve your problem as you may still have concurrency issues. Consider having something more sophisticated such as exploiting "embarrassingly parallel processing" such as having N processes creating N file chunks and finally having a single file joining all pieces together when everything is done.

Upvotes: 2

Naveen Vijay
Naveen Vijay

Reputation: 16492

As it is Amazon states that if you receive a success code then your S3 object is committed. Amazon also adds that there wouldn't be any dirty writes or overlapping inconsistency - you would read either of a fully committed write.

If you need more control you might be able to do it application like implementing a critical section.

It certainly makes sense to enable versioning the bucket so that you get to maintain all the writes and later you can specify which version as the latest.

You can also leverage the life cycle rules delete ( keep deleting ) the last n version to save cost.

Upvotes: 0

Related Questions