Dan
Dan

Reputation: 141

dbt pipeline call from Azure Data Factory using Azure Container Instance with a variable command

I'm looking to call dbt in a child pipeline from Azure Data Factory. Looking at the following project on GitHub it's been done before. https://gitlab.com/hashmapinc/oss/dbtonazure

What I would like to do though is pass a parametrised call from the Data Factory Web Activity so instead of having an Environment Variable in an Azure Containerised Instance and having to have a separate ACI for each dbt command I'd like to run I can pass a variable command to dbt.

e.g. Main pipeline runs and passes a variable "a". dbt runs dbt run --select a

Main pipeline runs and passes a variable "b". dbt runs dbt run --select b

I'm leaning to writing an Azure Function to accept incoming messages, where the message is the command.

Thanks, Dan

Upvotes: 3

Views: 1318

Answers (0)

Related Questions