CODEWITHSUNDEEP
azureazure-data-factory
VB_
VB_

Reputation: 45712

Azure Data Factory: common logic before/after all activities

Objective

I'm going to implement custom auditing on top of ADF pipelines. My ADF pipelines consist of activities of different types: ingesting, Databricks, loading results to data warehouse, etc. At the beggining and the end of each activity I would like to write some information to the auditing database.

The issue

I don't like to wrap each pipeline activity with extra auditing activies. I would like to do some kind of aspects/advices common for all activities. But I suppose that isn't a way ADF is supposed to be used.

Of course, it's easy to wrap custom Databricks/Python code with auditing, but how about ingesting/uploading activities?

Question

What is the best way to implement custom logic before/after all ADF pipeline activities?

P.S.

In Apache NiFi it's possible to access NiFi's logs and build a separate parallel pipeline that will parse and write logs to audit database. See the article. As far as I know, I can configure ADF to write logs to Azure Storage. Than read logs from there and write to audit database. But this sounds to me like a bad design. Any better options?

Upvotes: 1

Views: 597

Answers (1)

Bhushan
Bhushan

Reputation: 682

As far as I know, there is no direct way to do this apart from the approach you have mentioned i.e. to wrap each activity between pre and post processing logic and get it audited by logging your information to your sink, however I think it would be a good feature if activities natively starts supporting pre and post processing events in ADF itself and allow to invoke webhooks so that any pre and post processing data can be sent over to webhooks.

Further, You can submit an idea or suggestion to the team at feedback.azure.com/forums/270578-data-factory

Upvotes: 1

Related Questions