Action on error in Azure Machine Learning pipeline

Question

I have a published and scheduled pipeline running at regular intervals. Some times, the pipeline may fail (for example if the datastore is offline for maintenance). Is there a way to specify the scheduled pipeline to perform a certain action if the pipeline fails for any reason? Actions could be to send me an email, try to run again in a few hours later or invoke a webhook. As it is now, I have to manually check the status of our production pipeline at regular intervals, and this is sub-optimal for obvious reasons. I could of course instruct every script in my pipeline to perform certain actions if they fail for whatever reason, but it would be cleaner and easier to specify it globally for the pipeline schedule (or the pipeline itself).

Possible sub-optimal solutions could be:

• Setting up an Azure Logic App to invoke the pipeline
• Setting a cron job or Azure Scheduler
• Setting up a second Azure Machine Learning pipeline on a schedule that triggers the pipeline, monitors the output and performs relevant actions if errors are encountered

All the solutions above suffers from being convoluted and not very clean - surely there must exist a simple, clean solution for this problem?

Answer 1

This solution reads from the logs of your pipeline and let's you do something within a Logic App capability, I used it to email the team when a scheduled pipeline failed.

Steps:

1. Create Event Namespace and Event Hub
2. Create Service Bus Namespace and Service Bus Queue
3. Create a Stream Analytics Job using EventHub as Input and Service Bus Queue as Output
4. Create Logic App with a trigger to any event coming into the Service Bus Queue then, add an Outlook 360 send an email (v2) step
5. Create an Event Subscription inside ML Service that sends filtered events to the Event Hub
6. Start Stream Analytics Job

Two fundamental steps while creating the Event subscription:

• Subscribe to the 'Run Status Changed' event to get the log when a pipeline fails
• Use the advanced filters section to specify which pipeline you want to monitor (change 'deal-UAT' to your specific ml experiment), like this:

It looks like a lot of setup but it's super easy and quick to do, it would look something like this in the end: