CODEWITHSUNDEEP
azureazure-data-factory
Carlos Villadiego
Carlos Villadiego

Reputation: 87

Can datafactory start or stop self integration runtime?

I have a self integration runtime configured in a virtual machine on azure, because I need access on-premise database to load some information and this database can be accessed only throught VPN connection.

Is there a way to on/off the virtual machine when loading process is going to run (once a week) in order to optimize cost in the cloud ? because it not makes sense to me leave a vm billing at idle times.

Thank you

Upvotes: 1

Views: 2021

Answers (1)

Joseph Xu
Joseph Xu

Reputation: 6043

Updated:

  1. We can create Azure runbook, add PowerShell code to turn on/off the VM and call it in ADFv2 using Webhook according to this post.

  2. Create a trigger that runs a pipeline on a schedule. When creating a schedule trigger, you specify a schedule (start date, recurrence, end date etc.) for the trigger, and associate with a pipeline.

  3. At the start of the pipeline, you can use WebHook activity to start the vm, then copy data, at the end stop the vm. As follows: enter image description here

If you're looking to build data pipelines in Azure Data Factory, your cost will be split into two categories:

  • Data Factory Operations
  • Pipeline Orchestration and Execution

Data Factory Operations

  • Read/Write: Every time you create/edit/delete a pipeline activity or a Data Factory entity such as a dataset, linked service, integration runtime or trigger, it counts towards your Data Factory Operations cost. These are billed at $0.50 per 50,000 operations.

  • Monitoring: You can monitor each pipeline run and view the status for each individual activity. For each pipeline run, you can expect to retrieve one record for the pipeline and one record for each activity or trigger. For instance, you would be charged for 3 Monitoring activities if you debug a pipeline containing 2 activities. Monitoring activities are charged at $0.25 per 50,000 run records retrieved.

Pipeline Orchestration

  • Self Hosted Every time you run a pipeline, you are charged for every activity and trigger inside that pipeline that is executed at a rate of $1.50 per 1000 Activity runs.

As an example, executing a pipeline with a trigger and two activities would be charged as 3 Activity runs.

Pipeline Execution

Self hosted
Data movement : $0.10/hour
Pipeline activities : $0.002/hour
External activities : $0.0001/hour

An inactive pipeline is charged at $0.80 per month.

Summary:
When there are 30 days in a month and if your pipeline starts to run on a certain day and continues to run for 10 hours to move data. It will charge according to 10hx$0.10/h = $1. Pipelines that are inactive for an entire month are billed at the applicable "inactive pipeline" rate for the month as 29days/30days x $0.80 = $0.80. So your total cost is $1.8.

Upvotes: 1

Related Questions