CODEWITHSUNDEEP
pythongoogle-cloud-storageairflow
compassloire
compassloire

Reputation: 59

Uploading File to GCS Bucket in Airflow

I have an Airflow DAG with a variable set as a GCS bucket as follows:

bucket = 'customer_accounts'

When I run my Airflow DAG, everything works fine and a CSV file is uploaded into the customer_accounts bucket.

Now, I want to create a folder in the customer_accounts bucket and send all the CSV files from the DAG to the following folder in the bucket.

bucket = 'customer_accounts/canada'

However, when I try to run my DAG with bucket = 'customer_accounts/canada', I am getting an error message of an invalid bucket name. Do you know how to fix this issue?

Upvotes: 0

Views: 1210

Answers (1)

lander16
lander16

Reputation: 941

As mentioned in the comment, the bucket is still 'customer_accounts', Google Cloud Storage doesn't have the concept of folders in a traditional sense, they are part of the filename. You can learn more here https://cloud.google.com/storage/docs/gsutil/addlhelp/HowSubdirectoriesWork.

Upvotes: 1

Related Questions