Cloud Storage Appending and Logging

Question

What cloud service combination would allow for the storage of logs with N parts such that it is cheap and easy to:

1. Append new entries to the log
2. Request parts i -> j from the log, or at least parts i -> N
3. Have requests above be strongly consistent (i.e. reading the log after a user appends entry N + 1 always results in entry N + 1 being available)
4. Log entries are potentially quite large (10's to 100's of MB) so imagine that for a user that uploads a picture to some service the log entry I'm storing is [log and user data | picture content]. While I can happily store the log and user data separately from the picture content, when I retrieve the log I need to efficiently be able to get both sections.

Some research and thoughts I already have:

• S3/Google Cloud Storage do not seem to allow modifying object data, although Google allows for the compose function which might be useful.
• Google Cloud Storage and Windows Azure Storage are strongly consistent, S3 is not although might be able to use a DB to deal with consistency issues here.
• Microsoft Azure might be the best bet, since their Blobs seem to allow manipulating individual pages/blocks. However, I'm not sure whether using Blobs will truly satisfy the above requirements even if I'm OK with the caps on the maximum Blob size. For example, satisfying requirement 2 seems difficult without having a separate location (Blob/DB) that correlates log number with Blob number.

Answers suggesting that what I have come up with so far being close to optimal are fine, but I am wondering whether I have missed a strategy regarding these storage providers or another provider that I've entirely missed. Additionally, if I'm stuck using a DB for some data storage anyway it'd be interesting if minor tweaking could allow it to work efficiently across multiple cloud providers for cost and redundancy purposes, although that may just be a pipe dream.

Answer 1

You can consider storing every log entry as a separate blob in Azure Storage. With this solution;

1. Creates a new blob
2. Reads blobs i->j
3. Entire block blob content becomes readable as soon as it is committed with a single API call (Put Block List)
4. A single block blob can store up to 50K*4MB, so you can store the entire log entry in one blob.

Moreover, each blob has its own partition in Azure Storage. Blobs can therefore be distributed across many servers in order to scale out access to them.