Import Azure Monitor log data into Azure Data Factory

Question

Is it possible to connect Azure Data Factory to the Azure Monitor logs to extract the data? You can connect from Power BI as described here: https://learn.microsoft.com/en-us/azure/azure-monitor/platform/powerbi

But I want to be able to connect from Data Factory to the log.

Answer 1

Got is all working. I have done the following:

1. Create a Pipeline which contains 2 Web Activities, 1 For Each Loop & Call to stored procedure to insert the data
2. First Web Activity gets the bearer token
3. Second Web Activity calls the REST API GET and has a Header name Authorization which brings in the access_token for the first web activity Bearer {access_token}
4. Then A For Each Loop which I pass the output for the second Web Activity
5. Stored procedure Activity which passes in all my fields into an insert stored procedure

Finally that all worked. I had a lot of trouble using the Copy Activity so resorted to the For Each Loop and stored procedure call to insert each record from the output of the REST API call in the web activity.

I will post more detailed info once I get some sleep!

Answer 2

Partial Answer: I have been able to use 2 Web Activities in a pipeline. 1 which gets the bearer token and 2 which then uses the bearer token to carry out the GET command. BUT now the question is how can I use the Output from the Web Activity in a subsequent Copy Activity so I can load the data into SQL ????

Answer 3

Per my knowledge,there is no direct way which is similar to the PB way in the link you mentioned in your question in the ADF. Based on this document and ADF portal UI, we could store the log in three residences:

• Azure Storage Account.
• Event Hub.
• Log Analytics.

• For storage account ,you could access them in copy activity.
• For Event hub, maybe you could use Event Hub REST API in REST dataset and ADF Web Activity or you could get an idea of Azure Stream Analytics.

• For Log Analytics, you could use Log Analytics REST API in REST dataset and ADF Web Activity：

  POST https://api.loganalytics.io/v1/workspaces/DEMO_WORKSPACE/query

  X-Api-Key: DEMO_KEY
  Content-Type: application/json
  
  {
      "query": "AzureActivity | summarize count() by Category"
  }