Rails server running on multiple threads - How to make API calls to third parties thread-safe?

Question

I have a RoR application running on three different servers each running 5 threads. My application makes calls to a third party API. When making multiple concurrent requests I’m running into race conditions issues where a resource in that third party system might become unavailable as another request tries to access it. I’ve drawn this (very) quick diagram to illustrate.

So a user requests a resource to be deleted and shortly after another user requests a resource to be read, the read requests gets to the third party system faster for some reason or another so it returns it successfully while the other request is deleting it. Before my system receives the delete request and marks the resource as unavailable, it immediately sends a new request to edit that resource, which is now deleted from the third party system. This results in an error in my app where it might become out of sync. This is just an example and is not exactly what is happening (or the only thing that’s happening) but it illustrates the problem well.

I know there are ways to prevent race conditions like this from happening locally in ActiveRecord, but what is a good way to handle calls to third party API’s? Is there a way to process all the logic that handles the API calls in a synchronous manner without compromising the multithreaded nature of the servers?

Answer 1

When working with an external service, you cannot trust your own idea of the state their resources are in. Even if you sync up all your requests for your own system, something else can alter the resources. What if a user leaves an edit window open for a few hours, meanwhile the resource is deleted, and then submits an edit? Each request must be prepared to deal with resources in any possible state.

In this case all three requests must handle the case where the resource is not found: whether accessing, deleting, or editing a resource. If the user tries to edit and the resource does not exist, presumably you'll display a message like "sorry, that resource was not found or deleted".

Alternatively, rather than deleting resources you could deactivate them if possible. This allows you to tell the difference between a resource which does not exist, and one which was deleted. When a user tries to access or edit a deactivated resource you can give them a better error message ("resource was deleted" vs "resource not found") and perhaps offer to undelete the resource.

If available, you can use features such as webhooks and callbacks to be informed of changes and keep your own internal state of the external resources up to date. Even so these will have a delay and cannot be fully trusted.

Either way, each request cannot trust that they know the state of external resources.