Azure Functions Blob Trigger doesn't process all blobs

Question

I have around 35k blob files in my storage container. some of them are older before function was deployed some of them are after function app is deployed.

Here is my function signature

 public async Task Run([BlobTrigger("portal/{filePath}")]Stream blob, Uri uri,
            IDictionary<string, string> metadata, string filePath, ILogger log)
        {
         log.LogInformation($"{filePath}\n Size: {blob.Length} Bytes");

       // rest of the code
        }

My understanding is this function will continuously process all old and new files. But when I see that log on Azure portal, this is what happens.

• 2020-01-21T09:02:56.103 [Information] Executed 'watch-portal-blob' (Succeeded, Id=37cfaf56-3be2-4e6f-9e46-3558389daf4f)
• 2020-01-21T09:04:33 No new trace in the past 1 min(s).
• // more time stamps
• 2020-01-21T10:51:34 No new trace in the past 108 min(s).
• 2020-01-21T10:52:34 No new trace in the past 109 min(s).
• 2020-01-21T10:53:34 No new trace in the past 110 min(s).
• 2020-01-21T10:54:34 The log-streaming session has ended due to timeout 120 min(s).

It randomly processes few files and then go to idle state then again after few hours it process another 20 files. Can anyone point me to rightly configure blob trigger to process all existing blobs as well as new blobs without any break?

FYI: All my blobs are simple XML files, less than 20 KB each. All it does is read XML validate and store it in MongoDB. If I run this job on my local, in 1 minute minimum 50 files will be processed, so 30k files should be completed long back. On Friday Jan 17th there were around 35k files today on Jan 20th there are still 32k files (new additions are around 2k). That means it only processed 5k files in 3 days.

Answer 1

I do recommend to read the Trigger - blob receipts and the Trigger - polling

For your solution, the Azure Event Grid should be considered for a real-time push notification when the blob has been created/deleted in the blob storage.

Answer 2

Under the hood it uses some event notification in order to trigger your function. Which means your old blobs will not be triggered unless you change something on then (a metadata for example).

As an Alternative, you can create an Logic App and use the Azure Storage Connector + List blobs operation, then chain your current Azure Function in the workflow passing each blob.

https://learn.microsoft.com/en-us/connectors/azureblobconnector/#list-blobs