CODEWITHSUNDEEP
pythonpython-2.7flaskpython-rq
orangeandgrey
orangeandgrey

Reputation: 91

Python Flask: RQ Worker raising KeyError because of environment variable

I'm trying to setup a redis queue and a worker to process the queue with my flask app. I'm implementing this to handle a task that sends emails. I'm a little confused because it appears that the stack trace is saying that my 'APP_SETTINGS' environment variable is not set when it is in fact set.

Prior to starting up the app, redis or the worker, I set APP_SETTINGS:

export APP_SETTINGS="project.config.DevelopmentConfig"

However, when an item gets added to the queue, here's the stack trace:

17:00:00 *** Listening on default...
17:00:59 default: project.email.sendMailInBG(<flask_mail.Message object at 0x7fc930e1c3d0>) (aacf9546-5558-4db8-9232-5f36c25d521b)
17:01:00 KeyError: 'APP_SETTINGS'
Traceback (most recent call last):
  File "/home/tony/pyp-launch/venv/local/lib/python2.7/site-packages/rq/worker.py", line 588, in perform_job
    rv = job.perform()
  File "/home/tony/pyp-launch/venv/local/lib/python2.7/site-packages/rq/job.py", line 498, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "/home/tony/pyp-launch/venv/local/lib/python2.7/site-packages/rq/job.py", line 206, in func
    return import_attribute(self.func_name)
  File "/home/tony/pyp-launch/venv/local/lib/python2.7/site-packages/rq/utils.py", line 150, in import_attribute
    module = importlib.import_module(module_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/tony/pyp-launch/project/__init__.py", line 24, in <module>
    app.config.from_object(os.environ['APP_SETTINGS'])
  File "/home/tony/pyp-launch/venv/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'APP_SETTINGS'
Traceback (most recent call last):
  File "/home/tony/pyp-launch/venv/local/lib/python2.7/site-packages/rq/worker.py", line 588, in perform_job
    rv = job.perform()
  File "/home/tony/pyp-launch/venv/local/lib/python2.7/site-packages/rq/job.py", line 498, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "/home/tony/pyp-launch/venv/local/lib/python2.7/site-packages/rq/job.py", line 206, in func
    return import_attribute(self.func_name)
  File "/home/tony/pyp-launch/venv/local/lib/python2.7/site-packages/rq/utils.py", line 150, in import_attribute
    module = importlib.import_module(module_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/tony/pyp-launch/project/__init__.py", line 24, in <module>
    app.config.from_object(os.environ['APP_SETTINGS'])
  File "/home/tony/pyp-launch/venv/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'APP_SETTINGS'
17:01:00 Moving job to u'failed' queue
17:01:00 
17:01:00 *** Listening on default...

email.py

from flask.ext.mail import Message

from project import app, mail
from redis import Redis
from rq import use_connection, Queue

q = Queue(connection=Redis())


def send_email(to, subject, template, emailable):

    if emailable==True:   
        msg = Message(
            subject,
            recipients=[to],
            html=template,
            sender=app.config['MAIL_DEFAULT_SENDER']
        )
        q.enqueue(sendMailInBG, msg)

    else:
        print("no email sent, emailable set to: " + str(emailable))

def sendMailInBG(msgContent):

    with app.test_request_context():
        mail.send(msgContent)

worker.py

import os

import redis
from rq import Worker, Queue, Connection

listen = ['default']

redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')

conn = redis.from_url(redis_url)

if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(list(map(Queue, listen)))
        worker.work()

I'd really appreciate another set of eyes on this. I can't for the life of me figure out what's going on here.

Upvotes: 1

Views: 2547

Answers (1)

orangeandgrey
orangeandgrey

Reputation: 91

Thanks to the prompting of @danidee, I discovered that the environment variables need to be defined in each terminal. Hence, APP_SETTINGS was defined for the actual app, but not for the worker.

The solution was to set APP_SETTINGS in the worker terminal.

Upvotes: 2

Related Questions