CODEWITHSUNDEEP
databasemessage-queuepriority-queuetask-queue
Farzher
Farzher

Reputation: 14593

Persistent job queue?

Internet says using database for queues is an anti-pattern, and you should use (RabbitMQ or Beanstalked or etc)

But I want all requests stored. So I can later lookup how long they took, any failed attempts or errors or notes logged, who requested it and with what metadata, what was the end result, etc.

It looks like all the queue libraries don't have this option. You can't persist the data to allow you to query it later.

I want what those queues do, but with a "persist to database" option. Does this not exist? How do people deal with this? Do you use a queue library and copy over all request information into your database when the request finishes?

(the language/database I'm using is anything, whatever works best for this)

Upvotes: 3

Views: 3027

Answers (2)

pr-pal
pr-pal

Reputation: 3558

One can implement the task queue by themselves using a persistent backend (like database) to persist the tasks in queues. But the problem is, it may not scale well and also, it is always better to use a proven implementation instead of reinventing the wheel. These are tougher problems to solve and it is better to use the existent frameworks.

For instance, if you are implementing in Python, the typical choice is to use Celary with Redis/RabbitMQ backend.

Upvotes: 0

Alister Bulman
Alister Bulman

Reputation: 35169

If you want to log requests, and meta-data about how long they took etc, then do so - log it to the database when you know the relevant results, and run your analytic queries as you would expect to.

The reason to not be using the database as a temporary store is that under high traffic, the searching for, and locking of unprocessed jobs, and then updating or deleting them when they are complete, can take a great deal of effort. That is especially true if don't remove jobs from the active table, and so have to search ever more completed jobs to find those that have yet to be done.

Upvotes: 2

Related Questions