Reputation: 141
I'm looking to call dbt in a child pipeline from Azure Data Factory. Looking at the following project on GitHub it's been done before. https://gitlab.com/hashmapinc/oss/dbtonazure
What I would like to do though is pass a parametrised call from the Data Factory Web Activity so instead of having an Environment Variable in an Azure Containerised Instance and having to have a separate ACI for each dbt command I'd like to run I can pass a variable command to dbt.
e.g. Main pipeline runs and passes a variable "a". dbt runs dbt run --select a
Main pipeline runs and passes a variable "b". dbt runs dbt run --select b
I'm leaning to writing an Azure Function to accept incoming messages, where the message is the command.
Thanks, Dan
Upvotes: 3
Views: 1318