Exceptions in Microsoft.Azure.EventHubs.Processor Which resulted in CPU blocking

Question

We have a web job which uses "Microsoft.Azure.EventHubs.Processor". It started throwing below exceptions all of a sudden after continuously running without any issues more than 1 month. which resulted in CPU blockage. see the graph. We had to restart the web job to bring down the CPU.

• System.Net.Sockets.SocketException : An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
• Microsoft.Azure.Storage.StorageException: The lease ID specified did not match the lease ID for the blob.

Is there a way to handle the above exceptions. We didn't catch these exceptions even though we are handling exceptions in ProcessEventsAsync method. What are we missing here? How can we make sure it won't happen again?

Answer 1

Apparently the node that your web job was running on was out of available ports. Couple things you can do to investigate once you have a new repro:

• Take a process dump and check threads, thread pool, and socket instances.
• Check local ports, outgoing connections on the node. netstat command should list all ports in use.