CODEWITHSUNDEEP
azureazure-eventhubazure-stream-analytics
Shane
Shane

Reputation: 187

Minimize data transfer costs sending Event Hub/Stream Analytics Data across Azure regions

I have a service running in East US which collects request data. I need to get this data back to our central database running in North Europe. Currently, the web app running in East US is sending the info to an Event Hubs job in North Europe where a Stream Analytics job processes it and outputs to the db.

I want to minimize the Data transfer costs which, at scale, will pose a problem. I'm looking for ways to do this, such as:

But is there a benefit to moving the Event Hubs job to East US? Would the bandwidth going to the Event Hubs equal the bandwidth from the Event Hubs job to the Stream Analytics job? Does Stream Analytics pull the data in a way that is more efficient, perhaps by pulling compressed batches every x millisconds?

I can accept a delay of a few extra seconds if it means cutting down on the size of the data transfer.

Upvotes: 0

Views: 891

Answers (1)

Zhong Chen
Zhong Chen

Reputation: 46

I think it depends on whether you apply aggregation in your ASA job. A common usage of ASA is data volume reduction. For example, SQL server will not be able to handle the same level of throughput as Event Hub, so before writing to SQL server, people often compute windowed aggregates. If you apply the same pattern, it will make sense to run ASA job in East US, and write outputs with reduced volume to North Europe.

If you don't perform data reduction, the amount of data transfer will be similar whether you run ASA jobs in East US or North Europe. ASA uses Event Hub's AMQP protocol based client to retrieve events over TCP connection. It doesn't perform compression though. You can perform gzip compression on your payload at the sender size. ASA has an option to unzip the payload for processing.

Upvotes: 1

Related Questions