CODEWITHSUNDEEP
azure-eventgrid
eddyP23
eddyP23

Reputation: 6863

Azure Event Grid - logs of unmatched events

Recently we have been updating our Event Grid Subscriptions and because of an error in the migration, some of the subscriptions were not created. Dev errors happen, therefore the problem that I have here in mind is that Event Grid doesn't seem to offer any way to view unmatched events.

For example, matched events that don't get delivered (with some setup) end up in a storage account plain text. I tried turning on Event Grid Topic Logs hoping to find something useful there, but I did not find anything there apart from the numbers of messages matched/delivered/unmatched.

I was wondering whether I am missing something and would like to hear how others deal with this problem.

I created a feature proposal here.

Upvotes: 2

Views: 2065

Answers (1)

Roman Kiss
Roman Kiss

Reputation: 8265

The current version of the AEG doesn't support subscribing for UnmatchedEvents in the event source interest (storage, custom topic, etc.). All subscriptions are loosely decoupled and transparently from each others in the AEG eventing Fan-Out pattern.

Basically we can add this feature based on the complexity of the application eventing model.

  1. Simple solution using a Subscription Advanced Filtering for UnmatchedEvents. This solution is suitable for very small eventing model such as less than 25 subscriptions on the topic, see the Advanced Filtering limitations. The other limitation is a manually setting filtering for matched subscriptions and the unmatched subscription. The following is an example for this simple solution:

    enter image description here

The Advanced Filtering is setup for StringNotIn operation of the EventType. My example is using the valid EventTypes for matched subscriptions such as abc1, abc2, ... abc25, but it can be anything what the application eventing model requires with maximum 512 characters.

enter image description here

Basically, the Advanced Filtering is filtering (rejecting) for existing matched subscriptions in the EventTypes filter for example: 

 "includedEventTypes":[
      "abc1"
    ]
  1. The second solution allows to handle a "shadow filtering" in the audit subscriber. See the following screen snippet:

enter image description here

We have a Audit Subscription (no filtering on the topic) with an AzureFunction event handler. This function will filter an incoming message based on the knowledge of the all filter objects from existing subscriptions. If there is no match, the event message is stored to the storage queue with details of the filtering. Note, that the blob file can be automatically updated based on the resource (subscription) change. You can use a GET REST API to obtain all subscriptions on the specific topic, for example:

https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myRG/providers/Microsoft.EventGrid/topics/myTopic/providers/Microsoft.EventGrid/eventSubscriptions?api-version=2020-01-01-preview
  1. The third solution is suitable for large complex of the application eventing model, where are thousands subscriptions, multiple filtering, etc.:

enter image description here

As you can see, the above solution is using a pipeline stream to analyze a shadow AEG filtering. The Reference Data represents a real-time updated knowledge of the filter objects from the all subscriptions in the AEG model. The stream job will used it for specific event source (topic), etc. Note, that the job can run in the real-time or post-processing for auditing, troubleshooting, monitoring, learning process, etc.

Upvotes: 3

Related Questions