Emin TATLI
Emin TATLI

Reputation: 82

Insane amount of RAM usage with BullMQ + fetch

I have a small app that runs several requests(node-fetch) with in a queue with bullMQ.

And i have 3 worker threads (if i dont use this main event loop gets blocked and my whole app freezes.) (This is an another issue why main loop gets blocked with a few fetch requests ?)

queue A , queue B and queue C

all queues have same configuration

new Queue('a-queue', {
connection: {
    url: process.env.REDIS_URL,
    enableReadyCheck: true,
},
defaultJobOptions: {
    attempts: 5,
    removeOnComplete: true,
    removeOnFail: false,
},

worker for the queue (a,b,c all same):

const processor = pathToFileURL(__dirname + '/a_processor.js')
new Worker('a-queue', processor, {
    connection: {
        url: process.env.REDIS_URL,
        enableReadyCheck: true,
    },
    concurrency: 1,
    useWorkerThreads: true,
})

And a_processor.js is sending some requests with fetch and returns the values.

Also i have a setting for the global concurrency limit in index.js (start of the express app):

await queue.setGlobalConcurrency(5)

Upto this points it seems like it works as i expected (bullmq dashboard blue ones are active yellow ones are waiting): enter image description here

But after like a minute redis instance is crashing due high memory consumption:

enter image description here

Memory usage is like skyrocket when i run this app.And what is doing is just sending fetch requests (10-15 in paralel) and these requests are not like responsing mbs of data its only few kb of json each time. There is nothing much but this consumes so much memory idk how.Or someting else causing that idk. enter image description here

Here's the request method:

const headerGenerator = new HeaderGenerator()
const headers = headerGenerator.getHeaders()

const response = await fetch(`${domain}`, {
    headers: headers,
    agent: agentDcIp,
    signal: AbortSignal.timeout(15000) as any,
})

const body = await response.text()

return {
    body: body,
    ok: response.ok,
    status: response.status,
}

Any idea why this is happening and how to prevent it.

Upvotes: 0

Views: 273

Answers (0)

Related Questions