CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerygoogle-cloud-storagegsutil
kovalski1601
kovalski1601

Reputation: 1

Automate uploading files stored locally to Cloud Storage using gsutil

I'm new to GCP, I'm trying to build an ETL stream that will upload data from files to BigQuery. It seems to me that the best solution would be to use gsutil. The steps I see today are:

  1. (done) Downloading the .zip file from the SFTP server to the virtual machine
  2. (done) Unpacking the file
  3. Uploading files from VM to Cloud Storage
  4. (done) Automatically upload files from Cloud Storage to BigQuery

Steps 1 and 2 would be performed according to the schedule, but I would like step 3 to be event driven. So when files are copied to a specific folder, gsutil will send them to the specified bucket in Cloud Storage. Any ideas how can this be done?

Upvotes: 0

Views: 1673

Answers (1)

mhouglum
mhouglum

Reputation: 2593

Assuming you're running on a Linux VM, you might want to check out inotifywait, as mentioned in this question -- you can run this as a background process to try it out, e.g. bash /path/to/my/inotify/script.sh &, and then set it up as a daemon once you've tested it out and got something working to your liking.

Upvotes: 1

Related Questions