Remove old variants with Rails ActiveStorage

Question

I have something like a photo store application (Rails 5.2), which uses ActiveStorage to store the actual photos. All is well, I generate watermarked thumbnails for my users to see as Variants (originals are not public, and I actually need a few variants per image).

What if I have to change how I generate such variants? S3 storage is not cheap (number of images is in the tens of thousands ballpark), and ActiveStorage doesn't track variants at all - they are identified only by the hash of how exactly I generated them. If I change the variant hash (say change thumbnail size), new ones will be made ofc, but a lot of old blobs will remain in the storage unreferenced.

This same problem is also described here, but no solution.

What do you do with old variants? I want the variant storage to work kind of like a cache, especially that my users don't typically look at old images. If a variant is older than a given age, I want to simply remove it - but I have no way to do so. Or do I?

• I could look at actual blobs in the storage - but there is no difference, and no metadata. In S3, all of these files look the same, the key is just the hash of how it was made, and there is no relevant metadata.

• Hard-coding all previous variant-generation hashes to re-run on all previous images must not be the way to go.

• I could maybe patch something in ActiveStorage (Variation maybe?) to add metadata (indicate which blobs are variants and which are originals)? Then I could have backend jobs to remove old variants. Could this actually work? How would I make this compatible with future versions of ActiveStorage?

• Should I track variants myself? Why is it not an option in ActiveStorage then? This would also require changes to ActiveStorage classes, obviously.

• Can I store variants in a different storage (say another S3 bucket)? That would also solve the issue as I would then know which blobs are variants and could easily purge the old ones.

What do you people do with old, unnecessary variants?

Answer 1

I had some time to dig into ActiveStorage. Turns out, variants are stored with the variants/ prefix, which makes finding them a lot easier. :) It's a pity this isn't documented anywhere.

If I have a Pic that has_one_attached :image, I can delete all of its variants easily: pic.image.service.delete_prefixed("variants/#{pic.image.key}"). I will then have to call .processed on variants upon request though, to make sure they are recreated.

The only thing I still haven't figured out is how to find out variant age. I guess that info is lost, so I will have to manage that myself so that I know which variants have not been accessed lately and should be deleted.