CODEWITHSUNDEEP
c#azure-functions
Aaron
Aaron

Reputation: 652

Limited Scaling Azure Function

I have a scenario where I have a stream of URLs which I need to make a HTTP request against. I'll then download the data received and save it in Blob storage. I have to do this using Azure functions so that I'm only paying for the service when there are actually URLs to process.

However, the difficulty I'm having is conceiving of a way of triggering downloads through a limited number of proxies. Although I'm happy for the download function to scale out to the number of proxies I have available, I want each proxy to deal with each URL it receives in series. In other words, each proxy must be limited to downloading data from one URL at a time.

I considered having URLs in one queue and proxies in another queue and triggering a function when one of each is available, then pushing the used proxy back into the proxy queue, but functions can only take one trigger.

I also considered creating as many queues as there are proxies and distributing URLs between the queues, but I'm not sure how to limit the concurrency on each triggered function to one.

Anybody got an idea how to do this?

Upvotes: 0

Views: 56

Answers (1)

Aaron
Aaron

Reputation: 652

Okay, I found a way to do this via this post:

https://medium.com/@yuka1984/azure-functions-%E3%81%AE-singletonattribute%E3%81%A8mode%E3%83%97%E3%83%AD%E3%83%91%E3%83%86%E3%82%A3-bb728062198e

The answer is to add a [Singleton] attribute to the function.

However, according to this comment, you are spending money while your entities are awaiting processing:

https://github.com/Azure/azure-functions-host/issues/912#issuecomment-419608830

Upvotes: 1

Related Questions