CODEWITHSUNDEEP
node.jsherokudiscord.jsbull
Alaa
Alaa

Reputation: 1

How to use Bull, Throng, and Redis with Discord.js

I want to run long "scraping" processes on heroku server using discord as my front interface. I created a discord bot that runs the scraping when a command is used. all is working well. However, I want to use a queue system so that multiple users or even bulk usage will be possible.

Reading through heroku docs, I'm trying to add Bull to handle the queue system along with redis and throng. I got everything done and installed on my local machine for testing. I have the index file that listens to the discord command and fires off a job. and a worker file that handles the queue and the cluster using throng. Kicking off the job works fine. but the issue is how can i run the worker file! which will run the function that will execute the actual heavy work.

So, this is what I have tried so far. in my index file I listen to the discord command, add a new job to the queue, and reply to the user, last part is where I get the returned value when the job is done.

client.on('interactionCreate', async interaction => {
    if (!interaction.isChatInputCommand()) return
    const {commandName, options} = interaction
    if (commandName === 'views') {
        await workQueue.add('process', {text: 'Running...'})
        await interaction.reply(`processing... the results will be posted when done`)
    }
})
workQueue.on('global:completed', async (jobId, result) => {
    console.log(`Job completed with result ${result}`)
})

This is my worker file, where I create the start function as an entry point for throng *following the docs. I haven't added the actual heavy work method yet. so I'm just logging in and returning the job ID.

const throng = require('throng')
const Queue = require("bull")

const REDIS_URL = process.env.REDIS_URL || "redis://127.0.0.1:6379"
const workers = process.env.WEB_CONCURRENCY || 1
const maxJobsPerWorker = 50


function start() {
    // Connect to the named work queue
    const workQueue = new Queue('work', REDIS_URL)

    workQueue.process(maxJobsPerWorker, async job => {
        console.log("Im here")
        return {value: job.id}
    })
}

throng({workers, start})

How can I have the start function and get the returned value back so that I can send it back to the user? This is still a new concept to me, this might not be the right way to do it. so please feel free to correct me and guide me.

Upvotes: 0

Views: 380

Answers (1)

dberning
dberning

Reputation: 63

I believe you initialize your worker function inside your Procfile when you deploy to Heroku. So your Procfile may look something like:

web: node index.js
worker: node worker.js

To deploy, you will have to install node-foreman which handles your Procfile and update the start script in package.json file as follows:

start: nf start

Upvotes: 1

Related Questions