CODEWITHSUNDEEP
azureazure-functions
Marty Young
Marty Young

Reputation: 167

Am I expecting too much scalability from azure functions?

I have an azure function running on the consumption plan that gets triggered by a service bus topic then simply adds another message to a queue (so basically does no processing, just IO). The trigger topic is being filled at a rate of ~1000/second. I naively thought that the function could easily keep pace, but it is in fact hopelessly overwhelmed and the topic subscription fills very quickly. I have it running for hours so I suspect it is fully scaled out.

How much performance should I expect from functions? Is throughput on the order of thousands per second out of the question?

edit: I am regularly seeing this error in the logs:

The connection attempt lasted for a time span of 00:00:00. TCP error code 10013: An attempt was made to access a socket in a way forbidden by its access permissions.

Looks like functions don't play well with service bus at scale?

edit 2 for solution Replacing this binding: 

public static async Task Run(BrokeredMessage msgin, Binder binder, TraceWriter log)
{
 var collector = await binder.BindAsync<IAsyncCollector<BrokeredMessage>>(
                new ServiceBusAttribute("my-queue"));

...
}

with this:

public static IAsyncCollector<BrokeredMessage> collector;
public static async Task Run(BrokeredMessage msgin, Binder binder, TraceWriter log)
{
     collector = collector ?? await binder.BindAsync<IAsyncCollector<BrokeredMessage>>(
                new ServiceBusAttribute("my-queue"));


    ...
}

prevented socket exhaustion and the function was able to keep pace with the producer no problem.

Upvotes: 5

Views: 758

Answers (2)

Vukasin
Vukasin

Reputation: 90

I had the same issue in my Azure Function. The root cause was that I was exhausting all ports because I was creating a TCP connection inside Run method.

So when there were many parallel executions on the same machine, they all created connection, eventually exhausting all ports.

In that situation, connection was created as part of creation of ServiceBus client, and moving it to static variable resolved the issue.

Upvotes: 3

Matt Mason
Matt Mason

Reputation: 2726

Some questions:

  • Does your service bus connection string have manage rights? Manage rights are necessary for effective scaling at present.
  • Are you using input/output bindings, or are you managing the service bus client yourself?
  • Have you modified the service bus concurrency settings in host.json?

The error you're seeing is related to connection exhaustion, so somehow your function instance is running out of available connections.

If you want to get telemetry about functions scaling & throughput you can give this a shot: https://github.com/Azure/Azure-Functions/wiki/App-Insights-Early-Preview

Upvotes: 3

Related Questions