CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-glueamazon-kinesis-firehose
Prithvi Boinpally
Prithvi Boinpally

Reputation: 486

Delivering Files from AWS Kinesis Firehose to a Specific S3 Folder

Problem Context:
I have an AWS Kinesis Firehose delivery stream setup.
I also have an S3 bucket setup.
Within my S3 bucket I have a folder, lets call it: "folderX/".

Currently, it just sends it to a premade folder labeled "2021/" because the default scheme is to send to a folder corresponding to the current date/time.

Question: I want my Firehose delivery stream to send files into that specific folder. How do I do that?

Upvotes: 2

Views: 5079

Answers (1)

Balu Vyamajala
Balu Vyamajala

Reputation: 10333

Backup S3 bucket prefix in S3 Destination configuration is the spot to set a prefix.

From the Documentation for Backup S3 bucket prefix:

To use the default prefix for Amazon S3 objects, leave this option blank. Kinesis Data Firehose automatically uses a prefix in "YYYY/MM/dd/HH" UTC time format for delivered Amazon S3 objects. You can add to the start of this prefix

enter image description here

Couple more examples:

To get folderX/2021/3/17 we can use folderX/!{timestamp:YYYY}/!{timestamp:MM}/!{timestamp:dd}/

To get folderX/year=2021/month=3/day=17/hour=5/ we can use folderX/year=!{timestamp:YYYY}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/

Upvotes: 4

Related Questions