Azure Streaming Analytics Job Running Infinitely

Question

I am trying to work through Microsoft Learning - Implement a Data Streaming Solution with Azure Streaming Analytics. I am currently working through the exercise where we set up a job to to process, transform it with a query and then return the results.

The issue that I am having is that when I start my job - it just runs infinitely and doesn't finish. I am not receiving an error message. When I checked the documentation it said it may be an issue with the input, but I have went back to my input file and tested the data and it is visible.

Can anyone point me in the right direction to move on? Many thanks

Answer 1

actually, i had the same issue and after thinking about it for 5 minutes I remembered this is streaming job (continuous) and won' finish till you stop it. go to your output bob container and check if you have the file

Answer 2

I just finished this unit as well. The exercise says to start the pipeline "Now" which is the current time. This would cause the pipeline to start processing data only after the pipeline started running. I haven't tried it but if you put a new JSON document in that folder after the pipeline was up it would probably process fine.

I fixed it by specifying "Custom" and making sure I put a time before the example JSON input was put in the source blob container (an hour did it for me.)

After this change, once the pipeline was up the output counter incremented almost immediately and I could see the result in the correct directory.

Answer 3

Note: Azure Stream Analytics jobs runs forever until we manually stop the job.

Azure Stream Analytics jobs can be easily stopped or deleted through the Azure portal, Azure PowerShell, Azure SDK for .Net, or REST API. A Stream Analytics job cannot be recovered once it has been deleted.

MS learn tutorial - "Implement a Data Streaming Solution with Azure Streaming Analytics" - "Transform data by using Azure Stream Analytics", this module provides a basic example on how to create an Azure Stream Analytics job, set up an input, write a transformation query, and set up an output. You used Azure Storage as your source and destination, and you created a transformation query to produce some basic results. You also learned how to start a Stream Analytics job and view the job results.

In the summary section they have provided way to clean the resources in the resource group.

Clean up

To avoid unnecessary charges, you'll want to delete all the resources you created for this module. That includes the two storage accounts and the Stream Analytics job. You can delete these accounts individually, but it's easier to delete the resource group mslearn-streamanalytics:

1. Use the search field to find the resource group.

2. Select the group, and then select Delete resource group.

What is Azure Stream Analytics?

Azure Stream Analytics is a real-time analytics and complex event-processing engine that is designed to analyze and process high volumes of fast streaming data from multiple sources simultaneously. Patterns and relationships can be identified in information extracted from a number of input sources including devices, sensors, clickstreams, social media feeds, and applications. These patterns can be used to trigger actions and initiate workflows such as creating alerts, feeding information to a reporting tool, or storing transformed data for later use. Also, Stream Analytics is available on Azure IoT Edge runtime, enabling to process data on IoT devices.

The following scenarios are examples of when you can use Azure Stream Analytics:

• Analyze real-time telemetry streams from IoT devices
• Web logs/clickstream analytics
• Geospatial analytics for fleet management and driverless vehicles
• Remote monitoring and predictive maintenance of high value assets
• Real-time analytics on Point of Sale data for inventory control and anomaly detection

Reference: Azure Stream Analytics Overview