How do I safely delay a web response in ASP.net?

Question

I have a problem where when we initiate a REST resource from a third party (Twilio), the service responds so quickly, we don't have time to write our SID's to database. We can't tell the service to wait, as it returns the SID only when the service has initiated. The application itself can't hold the state, as there's no guarantee that the RESTful callback will reach the same instance of our application.

We've mitigated the problem by writing the SID's to a buffer table in the database, and we've tried some strategies for forcing the web response to wait, but using Thread.Sleep seems to be blocking other unrelated web responses and generally slowing down the server during peak load.

How can I gracefully ask a web response to hang on a minute while we check the database? Preferably without gumming up the whole server with blocked threads.

This is the code that initiates the service:

 private static void SendSMS(Shift_Offer so, Callout co,testdb2Entities5 db)
    {

        co.status = CalloutStatus.inprogress;
        db.SaveChanges();
        try
        {
            CallQueue cq = new CallQueue();
            cq.offer_id = so.shift_offer_id;
            cq.offer_finished = false;
            string ShMessage = getNewShiftMessage(so, co, db);
            so.offer_timestamp = DateTime.Now;
            string ServiceSID = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";

            var message = MessageResource.Create
                        (
                            body: ShMessage,
                            messagingServiceSid: ServiceSID,
                            to: new Twilio.Types.PhoneNumber(RCHStringHelpers.formatPhoneNumber(so.employee_phone_number)),
                            statusCallback: new Uri(TwilioCallBotController.SMSCallBackURL)
                        );
            cq.twilio_sid = message.Sid;
            db.CallQueues.Add(cq);
            db.SaveChanges();
            so.offer_status = ShiftOfferStatus.OfferInProgress;
            so.status = message.Status.ToString();
            so.twillio_sid = message.Sid;
            db.SaveChanges();

        }
        catch (SqlException e) //if we run into any problems here, release the lock to prevent stalling; 
                               //note to self - this should all be wrapped in a transaction and rolled back on error
        {
            Debug.WriteLine("Failure in CalloutManager.cs at method SendSMS: /n" +
                            "Callout Id: " + co.callout_id_pk + "/n"
                            + "Shift Offer Id: " + so.shift_offer_id + "/n"
                            + e.StackTrace);
            ResetCalloutStatus(co, db);
            ReleaseLock(co, db);
        }
        catch (Twilio.Exceptions.ApiException e) 
        {
            ReleaseLock(co, db);
            ResetCalloutStatus(co, db);
            Debug.WriteLine(e.Message + "/n" + e.StackTrace);
        }

    }

This is the code that responds:

        public ActionResult TwilioSMSCallback()
        {
            //invalid operation exception occurring here
            string sid = Request.Form["SmsSid"];
            string status = Request.Form["SmsStatus"];
            Shift_Offer shoffer;
            CallQueue cq = null;

            List<Shift_Offer> sho = db.Shift_Offers.Where(s => s.twillio_sid == sid).ToList();
            List<CallQueue> cqi = getCallQueueItems(sid, db);
            if (sho.Count > 0)
            {
                shoffer = sho.First();
                if (cqi.Count > 0)
                {
                    cq = cqi.First();
                }
            }
            else
            {
                if (cqi.Count > 0)
                {
                    cq = cqi.First();
                    shoffer = db.Shift_Offers.Where(x => x.shift_offer_id == cq.offer_id).ToList().First();
                }
                else
                {
                    return new Twilio.AspNet.Mvc.HttpStatusCodeResult(HttpStatusCode.NoContent);
                }
            }

            Callout co = db.Callouts.Where(s => s.callout_id_pk == shoffer.callout_id_fk).ToList().First();
            shoffer.status = status;
            if (status.Contains("accepted"))
            {
                shoffer.offer_timestamp = DateTime.Now;
                shoffer.offer_status = ShiftOfferStatus.SMSAccepted + " " + DateTime.Now;
            }
            else if (status.Contains("queued") || status.Contains("sending"))
            {
                shoffer.offer_timestamp = DateTime.Now;
                shoffer.offer_status = ShiftOfferStatus.SMSSent + " " + DateTime.Now;
            }
            else if (status.Contains("delivered") || status.Contains("sent"))
            {
                shoffer.offer_timestamp = DateTime.Now;
                shoffer.offer_status = ShiftOfferStatus.SMSDelivered + " " + DateTime.Now;
                setStatus(co);
                if (cq != null){
                    cq.offer_finished = true;
                }
                CalloutManager.ReleaseLock(co, db);
            }
            else if (status.Contains("undelivered"))
            {
                shoffer.offer_status = ShiftOfferStatus.Failed + " " + DateTime.Now;
                setStatus(co);
                if (cq != null){
                    cq.offer_finished = true;
                }
                CalloutManager.ReleaseLock(co, db);
            }
            else if (status.Contains("failed"))
            {
                shoffer.offer_status = ShiftOfferStatus.Failed + " " + DateTime.Now;
                setStatus(co);
                if (cq != null){
                    cq.offer_finished = true;
                }
                cq.offer_finished = true;
                CalloutManager.ReleaseLock(co, db);
            }
            db.SaveChanges();
            return new Twilio.AspNet.Mvc.HttpStatusCodeResult(HttpStatusCode.OK);
        }

This is the code that delays:

public static List<CallQueue> getCallQueueItems(string twilioSID, testdb2Entities5 db)
    {
        List<CallQueue> cqItems = new List<CallQueue>();
        int retryCount = 0;
        while (retryCount < 100)
        {
            cqItems = db.CallQueues.Where(x => x.twilio_sid == twilioSID).ToList();
            if (cqItems.Count > 0)
            {
                return cqItems;
            }
            Thread.Sleep(100);
            retryCount++;
        }
        return cqItems;
    }

Answer 1

Good APIs™ let the consumer specify an ID that they want their message to be associated with. I have never used Twilio myself, but I have read their API Reference for Creating a Message Resource now, and sadly it seems like they don't provide a parameter for this. But there's still hope!

Potential solution (preferred)

Even though there isn't an explicit parameter for it, maybe you can specify slightly different callback URLs for each message that you create? Assuming that your CallQueue entities have a unique Id property, you could let the callback URL for each message contain a query string parameter specifying this ID. Then you can handle the callbacks without knowing the Message Sid.

To make this work, you would reorder things in the SendSMS method so that you save the CallQueue entity before invoking the Twilio API:

db.CallQueues.Add(cq);
db.SaveChanges();

string queryStringParameter = "?cq_id=" + cq.id;
string callbackUrl = TwilioCallBotController.SMSCallBackURL + queryStringParameter;

var message = MessageResource.Create
(
    [...]
    statusCallback: new Uri(callbackUrl)
);

You would also modify the callback handler TwilioSMSCallback so that it looks up the CallQueue entity by its ID, which it retrieves from the cq_id querystring parameter.

Solution that is almost guaranteed to work (but requires more work)

Some cloud services only allow callback URLs that exactly match one of the entries in a pre-configured list. For such services, the approach with varying callback URLs won't work. If this is the case for Twilio, then you should be able to solve your problem using the following idea.

Compared to the other approach, this one requires bigger changes to your code, so I will only give a brief description and let you work out the details.

The idea is to make TwilioSMSCallback method work even if the CallQueue entity doesn't yet exist in the database:

• If there is no matching CallQueue entity in the database, TwilioSMSCallback should just store the received message status update in a new entity type MessageStatusUpdate, so that it can be dealt with later.

• "Later" is at the very end of SendSMS: Here, you would add code to fetch and process any unhandled MessageStatusUpdate entities with matching twilio_sid.

• The code that actually processes the message status update (updating the associated Shift_Offer, etc.) should be moved away from TwilioSMSCallback and be placed in a separate method that can also be called from the new code at the end of SendSMS.

With this approach, you would also have to introduce some kind of locking mechanism to avoid race conditions between multiple threads/processes trying to process updates for the same twilio_sid.

Answer 2

Async/await can help you not to block your threads.

You can try await Task.Delay(...).ConfigureAwait(false) instead of Thread.Sleep()

UPDATE

I see you have some long running logic in TwilioSMSCallback and I believe this callback should be executed as fast as possible as it's coming from Twilio services (there can be penalties).

I suggest you to move your sms status processing logic to the end of SendSMS method and poll database there with async/await until you get the sms status. This however will keep SendSMS request active on the caller side, so better to have separate service which will poll database and call your api when something changes.

Answer 3

You really should not delay a RESTful call. Make it a 2-step operation, one for starting it, and one for getting the state. The latter you may call more than once, until the operation has safely completed, is light-weight and allows also for a progress indicator or state feedback to the caller, if you desire so.