Rails worker written in Javascript?

Question

We have a Node application, and a portion of the Node code runs as a background job. The job is using BullMQ, which is written in JavaScript and uses Redis under the hood.

We are in the process of porting some Node application to Rails. But, I'd like to keep the background job, written in Node as it is.

Is it possible to have:

1. A route in Rails, accept a request, schedule a job and push it to a Redis queue?
2. Have the job processed by a worker written in Node/JavaScript?
3. Have Rails listen to the Queue (Websockets or long polling) and send result to browser when complete?

Another words is there an language independent Redis / Queue system?

                                                       +-------------+
                                                       |             |
                                                       |             |
                                                       | REDIS QUEUE |
                                      +----------------+             |
                                      |                |             |
                                      |                |             +^--+
                                      |                +---------------+ |
     +---------+                      |                       ^        | |
     |         |              +-------v--+                    |        | |
     | browser +--------------+          +--------------------+        | |
     |         <--------------+  Rails   |                             | |
     +---------+              |          |                             | |
                              +----------+                   +---------v----+
                                                             |              |
                                                             |              |
                                                             |  NODE worker |
                                                             |              |
                                                             |              |
                                                             +--------------+

I am more familiar with Jobs/Queue in Node than in Rails. I saw a lot of options like SideKick and Resque, but it seems that the workers are mostly written in Ruby.

Answer 1

To answer the 3 questions:

1. a route in Rails, accept a request, schedule a job and push it to a Redis queue

Yes, certainly possible to push a new job's attributes to the redis queue. You just build your arguments (probably serialize them as JSON) and push to the queue with Redis' rpush (can use Ruby's Redis gem for this).

2. have the job processed by a worker written in Node/Javascript

Yes, there is fundamentally nothing about Redis that says the writer needs to be written in the same language as the reader - if the Rails app serializes the input to JSON, the Node app can pop that JSON and process it

3. Have Rails listen to the Queue and send result to browser when complete

This is the trickier part. The key thing to understand is that a request is a synchronous process but the whole background job system is async - hypothetically you could just tell your controller to sleep and check for confirmation on a loop, but this is inefficient for 2 reasons - first, it's not really necessary to query redis for the confirmation on a loop (since Redis has a "subscribe" feature) and second, you're going to have some wasted memory if you make your controllers sleep.

A better way to do it would be to use websockets. To schedule the job, you can either use a regular controller or an inbound websocket message handler - in either case it will schedule the job and immediately respond "the job is enqueued". Then, separately in the Rails app (outside of the controller methods) you can set up a Redis subscription to the "job done" queue. When it receives a message, it will find the websocket connection to the relevant client and send an outbound message that the Job was processed.