Implement web hooks for bulk actions

Question

I'm looking to implement web hooks as part of an applications REST API.

Initially we are looking to implement a mechanism for API consumers to register for entity updated events. So if an entity changes, we invoke all the webhooks that have registered for change events on that entity (and as part of the registration process, API consumers can include additional filter criteria to ensure they only receive callbacks for the subset of entities they are interested in).

Now - this works well for user initiated changes, that will trickle in slowly, but I'm concerned about how best to deal with this when a flood of changes occur - for example as part of a bulk action performed in the UI, or a flood of changes occurring from API consumers.

So far I have considered:

• Just doing a callback for each entity, using some kind of async pool - problem I see here is scale and potentially causing harm to those applications subscribe to the web hooks.
• Queuing the records of change per webhook registration, over say a 10 second window, and then pushing a single webhook notification to the subscribe with a list of the affected entities - problem I see here is that we introduce unnecessary latency between the action and the webhook, when events are just trickling in - also this introduces some overhead and complexity, especially if implementing this in a web-farm scenario.
• Actually exposing the bulk actions as web hooks - so if a bulk delete was performed, a consumer would subscribe to this. So setting up a hook for individual entity changes would not receive any entity change events for bulk updates/deletes etc. - they would have to handle this via the bulk action web hooks.

As some extra details - a bulk action in this application would likely encompass anywhere between 10 and around 100,000 entities.

Has anyone implemented web hooks for bulk actions that can shed any light on how they decided to implement this?

Answer 1

We recently implemented web hooks as part of our rest API and our major concern was also about bulk action. In our case it is bulk import, where a user can import records in a excel or csv file through our web app.

Our import process is designed in such a manner that the entire process runs in a transaction. So if it fails we simply roll back and do nothing, and if it successfully completes we post one web hook notification to the subscribed client with one huge body with info about the affected entities.

Now I am not sure how bulk actions take place in your application. If its within a transaction then you've got no issue. Other wise I'd recommend keeping the single and bulk action web hooks separate. I guess this is one of the drawbacks of using webhooks, you could bring down your subscriber with back to back POST requests.