Race condition with webhooks when updating a Stripe Connect account and my db, that cannot be resolved with concurrency exception

Question

Scenario - I'm updating a Stripe Connect(bank) account, that has a payout status of disabled, by providing it with an additional identifying document for verification. This will enable the payout status, so that the user can receive payouts. When Stripe receives the document from my server it sends 2 "account.updated" webhooks that I read and save the data from in my db. One of these two webhooks has the updated payout enabled status of true, the other webhook has a payout status of false.

PROBLEM1 - There is no guarantee that I receive the webhooks in the order in which they are sent. Stripe order of events So in some cases I might receive the event updated webhook with the disabled payout status 1st and the event updated webhook with the enabled payout status 2nd.

PROBLEM2 - Even if I receive the webhooks in order (payouts disabled 1st, payouts enabled 2nd) my server might process them at different times (depending on load) and save them out of order from the order with which I received them. Once again leaving my user seeing a disabled account, even when it's enabled.

FYI - From what I can see from my testing so far, Stripe always seems to generate and send the account updated webhook in the correct order (disabled 1st, enabled 2nd). Here are 2 examples.

EXAMPLE 1 - From Stripe dashboard, notice the timestamp difference

Event: evt_1JfsTQQaRtdb8QPYdufePsBa => "payouts_enabled": true

Event: evt_1JfsTQQaRtdb8QPYdh17AreO => "payouts_enabled": false

EXAMPLE 2 - From Stripe dashboard, notice the timestamp is the same

Event: evt_1JgaR7QgKzBv3Rh9Dx6Xq3W9 => "payouts_enabled": true

Event: evt_1JgaR7QgKzBv3Rh996DT3gBC => "payouts_enabled": false

GOAL - Making sure that the most current account info is updated on my side from the webhooks that I receive from Stripe, regardless of the order with which they are processed or received. So that I don't end up with an account payout status of disabled to display to the user, when the payout status is true.

What I've tried - Solving problem 1 and 2 above seemed simple enough. I implemented optomistic concurrency using a DbUpdateConcurrencyException, where I looked at the time stamp (stripeEvent.Created) of the Stripe event and only updated my DB when the concurrency exception is thrown and if the Stripe event timestamp I receive is later than what's in the current table row column "EventCreatedDate".

ex.

    [HttpPost("stripe")]
    public async Task<ActionResult> StripeWebhook()
    {
        var json = await new StreamReader(HttpContext.Request.Body).ReadToEndAsync();

        try
        {
            Event stripeEvent = EventUtility.ConstructEvent(json, Request.Headers["Stripe-Signature"], WhSecret);

            // Handle the events
            if (stripeEvent.Type == Events.AccountUpdated)
            {
                var stripeAccount = stripeEvent.Data.Object as Stripe.Account;
                var spec = new StripeAccountWithTypeSpecification(stripeAccount.Id);
                var dbEntity = await _unitOfWork.Repository<StripeAccount>().GetEntityWithSpec(spec);
                if (dbEntity == null) 
                    return BadRequest();
                // only update the db row if the Stripe account created timestamp is the same or later than what he have in the DB (EventCreatedDate)
                if (dbEntity.EventCreatedDate.CompareTo(stripeAccount.Created) <= 0) {
                    _mapper.Map(stripeAccount, dbEntity);
                    dbEntity.EventCreatedDate = stripeEvent.Created;
                    _unitOfWork.Repository<StripeAccount>().Update(dbEntity);
                    try
                    {
                        var success = await _unitOfWork.Complete();
                    }
                    catch (DbUpdateConcurrencyException ex)
                    {
                        // Get the current entity values we are saving and the values in the database
                        var entry = ex.Entries.Single();
                        var currentValues = entry.CurrentValues;
                        var entityDate = currentValues.GetValue<DateTime>("EventCreatedDate");
                        
                        var databaseValues = entry.GetDatabaseValues();
                        var databaseDate = databaseValues.GetValue<DateTime>("EventCreatedDate");

                        // If the Stripe created datetime timestamp (EventCreatedDate) in the database is in the past or equal to the timestamp we are trying to insert then we insert it.
                        // This will ensure only the most recently sent events get updated in the DB, if we experienced a concurrency exception
                        if (databaseDate.CompareTo(entityDate) <= 0) {
                            // Update the values of the entity that failed to save from the store
                            ex.Entries.Single().Reload();
                        }
                        _logger.LogInformation("Concurrency Exception Thrown");
                    }
                }
            }
            else if (stripeEvent.Type == Events.AccountApplicationDeauthorized)
            {
                var application = stripeEvent.Data.Object as Stripe.Application;
                _logger.LogInformation("Account application deauthorized id: {0}: ", application.Id);
            }
            else
            {
                // Unexpected event type
                _logger.LogInformation("Unhandled event type: {0}", stripeEvent.Type);
            }
            return Ok();
        }
        catch (StripeException ex)
        {
            _logger.LogWarning("Stripe Exception: {0}, {1}", ex.Message, ex);
            return BadRequest();
        }
    }

PROBLEM WITH MY SOLUTION - The problem is that if you look at the Stripe dashboard event timestamps shown from example 2, they're the same! So I could end up with a scenario where I process the disabled payout webhook 2nd and because it has the same timestamp as the webhook event with a payout status of enabled. It will overwrite the payout status true event, leaving me with a payout status of disabled!!! So I can't rely or use the Stripe event timestamp to check which events are sent first.

QUESTION - I'm currently looking to see if there are "Stripe.Event" properties that can be used to determine the most current event sent to me. Does anyone have any advice for me on what I can do here?

Answer 1

I struggled with this concurrency issue for a while too. The only difference is that it affected subscriptions, but the logic is the exact same. In the end, here's what I did :

I left my original logic that handles every incoming event right away. Most of the time, there are no race conditions. This way, it provides instant feedback in our database for almost every customer.

On top of that, I now use our queue to start a job which will make sure the subscription is properly synced. It's a very simple implementation where I delay a SyncStripeSubscription job by one minute. Its only job data is the stripe subscription id. It then fetches the stripe subscription, which has supposedly been properly updated on Stripe's end after this little delay, and updates the database accordingly just in case the race condition happened.

While a bit more convoluted, this logic ensures that, at the very worst, our database will be properly synced after one minute.