CODEWITHSUNDEEP
node.jsapiqueueamqpworker
RachelD
RachelD

Reputation: 4089

API with Work Queue Design Pattern

I am building an API that is connected to a work queue and I'm having trouble with the structure. What I'm looking for is a design pattern for a worker queue that is interfaced via a API.

Details:

I'm using a Node.js server and Express to create an API that takes a request and returns JSON. These request can take a long time to process (very data intensive) so this is why we use a queuing system (RabbitMQ).

So for example lets say I send a request to the API that will take 15 min to process. The Express API formats the request and puts it in a RabbitMQ (AMQP) queue. The next available worker takes the request off the queue and starts to process it. After its done (in this case 15 min) it saves the data into a MongoDB. .... now what .....

My issue is, how do I get the finished data back to the caller of the API? The caller is a completely separate program that contacts the API via something like an Ajax request.

The worker will save the processed data into a database but I have no way to push back to the original calling program.

Does anyone have any API with a work queue resources?

Please and thank you.

Upvotes: 3

Views: 1235

Answers (1)

Weston
Weston

Reputation: 2751

On the initiating call by the client you should return to the client a task identifier that will persist with the data all the way to MongoDB.

You can then provide an additional API method for the client to check the task's status. This method should take a single parameter, the task identifier, and check if a document with that identifier has made in into your collection in MongoDB. Return false if it doesn't exist yet, true when it does.

The client will have to repeatedly poll (but maybe at a 1 minute interval) the task status API method until it returns true.

Upvotes: 3

Related Questions