CODEWITHSUNDEEP
pythonbashamazon-s3airflowsoda
Dynastywarriorlord07
Dynastywarriorlord07

Reputation: 27

How to run a soda command when the checks and configuration yml files are on an S3 location?

I am trying to execute the SODA command to run a checks.yml file which contains sql queires comparing counts between the source database and target data warehouse on snowflake. I am trying to find a way where I can move my configuration and checks files onto a location on S3 and still be able to execute the command based on the files in the S3 location. Is there a way to do this?

Also, is it possible to incorporate this using airflow?

soda command: soda scan -d ds -c {SODA_PATH}/config.yml {SODA_PATH}/checks.yml

ariflow operator right now:soda_scan = bashOperator(task_id="dq_test", bash_command = f"soda scan -d ds -c {SODA_PATH}/config.yml {SODA_PATH}/checks.yml")

Is there an airflow operator which will allow me to get the files directly from S3?

Thanks

Upvotes: 1

Views: 207

Answers (0)

Related Questions