CODEWITHSUNDEEP
amazon-web-servicesaws-glueamazon-kinesisamazon-kinesis-firehose
coderz
coderz

Reputation: 4989

Aggregate multiple S3 files into one file

I enabled a Firehose stream to write data to S3. Firehose puts data into S3 file at max interval of 900s. This means around 100 files will be created within one day, which is an overhead for users to manually download.

Is there a solution to automatically aggregate multiple S3 files into one file at certain frequency (such as daily)?

Upvotes: 1

Views: 1582

Answers (1)

Marcin
Marcin

Reputation: 238071

There is no auto-of-the box service for that as this is a use-case specific problem. However, in AWS blog titled

they provide some guideline on how to do it effectively. In this blog they say that you could treat those files as parts of multi-upload and have S3 join them automatically for you.

I am not sure if this solution is applicable to your use-case, but its probably worth considering.

Otherwise, you have to download these objects, join them yourself (e.g. zip them), and re-upload joined folder. This could be automated, e.g. using instance or container that you run automatically once a day.

Upvotes: 1

Related Questions