Deleting Blobstore orphans

Question

What is the most efficient way to delete orphan blobs from a Blobstore?

App functionality & scope:

• A (logged-in) user wants to create a post containing some normal datastore fields (e.g. name, surname, comments) and blobs (images).
• In addition, the blobs are uploaded asynchronously before the resto of the data is sent via a POST
  • This leaves a good chance of having orphans as, for example, a user may upload images but not complete the form for one reason or another. This issue would be minimized by not using an asynchronous upload of the blobs before sending the rest of the data, however, this issue would still be there on a smaller scale.

Possible, yet inefficient solutions:

• Whenever a post is completed (i.e. the rest of the data is sent), you add the blob keys to a table of "used blobs". Then, you can run a cron every so often and compare all of the blobs with the table of "used blobs". Those that have been uploaded over an hour ago yet are still "not used" are deleted.
  • My understanding is that running through a list of potentially hundreds of thousands of blob keys and comparing it with another table of hundreds of thousands of "used blob keys" is very inefficient.

Is there any better way of doing this? I've searched for similar posts yet I couldn't find any mentioning efficient solutions.

Thanks in advance!

Answer 1

Use Drafts! Save as draft after each upload. Then dont do the cleaning! Let the user for himself chose to wipe out.

If you're planning on posts in a Facebook style use drafts either or make it private. Why bother deleting users' data?

Answer 2

Thank for the comments. However, I understood those solutions well, I find them too inefficient. Querying thousands of entries for those that are flagged as "unused" is not ideal.

I believe I have come up with a better way and would like to hear your thoughts on it:

When a blob is saved, immediately a deferred task is created to delete the same blob in an hour’s time. If the post is created and saved, the deferred task is deleted, thus the blob will not be deleted in an hour’s time.

I believe this saves you from having to query thousands of entries every single hour.

What are your thoughts on this solution?

Answer 3

You can create an entity that links blobs to users. When a user uploads a blob, you immediately create a new record with the blob id, user id (or post id), and time created. When a user submits a post, you add a flag to this entity, indicating that a blob is used.

Now your cron job needs to fetch all entities of this kind where a flag is not equal to "true" and time created is more one hour ago. Moreover, you can fetch keys only, which is a more efficient operation that fetching full entities.

Answer 4

A blob has also a filename. After the post you can delete all the old blobs with the same filename. The duplicates to delete must have the same owner or do not have an owner. You also have to delete the blobs which do not have an owner.

Here is an example to delete the duplicates, after an upload.

blobs = blobstore.BlobInfo.gql("WHERE filename = '%s'" %(filename))
for blob in blobs :
    if blob.key() != userdata.blob_ref.key()  : blob.delete()

To clean up "not used" blobs, you can schedule a task after every upload, to run after an hour.