CODEWITHSUNDEEP
pythonredisrq
sc0urge
sc0urge

Reputation: 41

Redis Queue task is queued but never executed

I am having issues getting RQ-python to run. Like in the example of the documentation (https://python-rq.org) I have a function in an external file

def createAndSaveIndex(url_list, index_path):
    print("--------------------------------------started task------------------------------------")
    index = indexFromURLList(url_list=url_list)
    index.save_local(index_path)
    return "Im done!"

which I import into my main file and use in the queue:

from redis import Redis
conn = Redis()
q = Queue(connection=conn)

job = q.enqueue(f=createAndSaveIndex,
                args=(["amazon.com"], dirname+"/myIndex/"))   # how long to hold onto the result

print(job.id)

The job is created and I get the job Id, however not even the print statement is executed and job.is_finished always returns false. I am on MacOs and have redis installed thorugh homebrew. I called redis-server and using it through the terminal works too? Does anyone have an idea what I could have done wrong?enter image description here

Tried the different examples I found online, none of which worked in my case. Tried using the function inside and outside the file.

Edit I forgot to check and mention the worker, turns out it is actually throwing an error:

File "/Users/-/micromamba/envs/flask/lib/python3.11/site-packages/rq/utils.py", line 107, in import_attribute
    return __builtins__[name]
           ~~~~~~~~~~~~^^^^^^
KeyError: 'backend_index.createAndSaveIndex'

as well as:

File "/Users/-/micromamba/envs/flask/lib/python3.11/site-packages/rq/utils.py", line 109, in import_attribute
    raise ValueError('Invalid attribute name: %s' % name)
ValueError: Invalid attribute name: backend_index.createAndSaveIndex

backend_index is the file where my function is located and createAndSaveIndex the name of the function.

Edit I first had both of my python files in the root directory I was told that the function must come from a module and not just a file, so now this is my project structure, however nothing changed for me:

project/
├── main.py
└── indexCreation/
    ├── __init__.py
    └── createIndex.py (was named backend_index)

This is the content of my init.py

from .createIndex import createAndSaveIndex

Solution After trying to recreate the error on another computer I finally got it to work: Turns it out the issue was the kind of terminal I used. When creating the redis-server and initializing the rq worker I used my macOs Terminal and to run the code I used the VSCode built in terminal (Which I thought was the same since I was using the same venv). Now If you do everything in the integrated vscode terminal it works with no issue. Thank you very much @รยקคгรђשค for helping me find the issue.

Actual solution When working with rq again I encountered the error once again, but for another reason. Also make sure when calling rq worker, that your working directory is also the one where your imported module is located.

Upvotes: 3

Views: 1145

Answers (1)

sc0urge
sc0urge

Reputation: 41

Solution: After trying to recreate the error on another computer I finally got it to work: Turns it out the issue was the kind of terminal I used. When creating the redis-server and initializing the rq worker I used my macOs Terminal and to run the code I used the VSCode built in terminal (Which I thought was the same since I was using the same venv). Now If you do everything in the integrated vscode terminal it works with no issue. Thank you very much @รยקคгรђשค for helping me find the issue.

Upvotes: 0

Related Questions