Celery / Heroku - delay() does nothing when ran in the background using Heroku run python

Question

I am attempting to add background workers using Celery to automatically run scripts each day to update the information my website provides. I've integrated Celery with Django and Heroku, and I can import the module and use the function, but it freezes when I use the add.delay()command until I press Ctrl+C to cancel the command. I am using celery 4.1 Here is how I run the commands:

heroku ps:scale worker=1
heroku run python
>>>from Buylist.tasks import *
>>>add(2,3)  
>>>5
>>>add.delay(2,3)  
#-- Freezes until I press Control+C

If you could help me figure out where my settings are misconfigured, that would be great. I'm testing atm. the tasks.py is the sample code to get a working example, and then I'll move on to figuring out CELERY_BEAT settings

project/celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery


# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

app = Celery('project')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
app.conf.update(BROKER_URL=os.environ['REDIS_URL'],
                CELERY_RESULT_BACKEND=os.environ['REDIS_URL'])

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

Buylist/tasks # Buylist is the single app in my root directory

# Create your tasks here
from __future__ import absolute_import, unicode_literals
from celery import shared_task
from celery import Celery
from celery.schedules import crontab


@shared_task
def test(arg):
    print(arg)

@shared_task

def add(x, y):

    return x + y


@shared_task

def mul(x, y):

    return x * y


@shared_task

def xsum(numbers):

    return sum(numbers)

project/settings.py

from __future__ import absolute_import, unicode_literals
import dj_database_url


"""
Django settings for project project.

Generated by 'django-admin startproject' using Django 2.0.

For more information on this file, see
https://docs.djangoproject.com/en/2.0/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/2.0/ref/settings/
"""

import os
import django_heroku
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))



# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.0/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
#Read secret key from a file

SECRET_KEY = 'KEY'



# SECURITY WARNING: don't run with debug turned on in production!
#DEBUG = True
DEBUG = bool( os.environ.get('DJANGO_DEBUG', True) )

ALLOWED_HOSTS = [
                'shrouded-ocean-19461.herokuapp.com', 'localhost', '127.0.0.1',

]

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_celery_results',
    'Buylist',
    #'django_q',

]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'project.urls'

PROJECT_DIR = os.path.dirname(__file__)
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            os.path.join(PROJECT_DIR, "templates"),
        ],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'project.wsgi.application'


# Database
# https://docs.djangoproject.com/en/2.0/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}


# Password validation
# https://docs.djangoproject.com/en/2.0/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/2.0/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.0/howto/static-files/

# Heroku: Update database configuration from $DATABASE_URL.

db_from_env = dj_database_url.config(conn_max_age=500)
DATABASES['default'].update(db_from_env)

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.10/howto/static-files/

# The absolute path to the directory where collectstatic will collect static files for deployment.
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

# The URL tos use when referring to static files (where they will be served from)
STATIC_URL = '/static/'

# Simplified static file serving.
# https://warehouse.python.org/project/whitenoise/
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

#CELERY_RESULT_BACKEND = 'django-db'

project/init.py

from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ['celery_app']

Here is my procfile I have the Heroku Redis app installed on Heroku's end

web: gunicorn project.wsgi --log-file -
worker: celery worker --app=tasks.app

Procfile is located in the main Git root directory. Let me know if you need more info and I can provide it! Thanks a bunch!

Answer 1

The procfile for a single worker should look something like this

worker: celery -A <folder containing celery.py> worker -l info

My broker url was also not configured correctly. Setup your broker on heroku, and click on it from the overview page once you have added it to your project. In my case I used CloudAMQP - RabbitMQ manager. Once you click on it, there will be information about your broker url. Password, username, url etc. You want to copy the url, and in your Django app's settings.py or somewhere where you are configuring your Celery config files, you want to set your broker url. Mine looks like this in settings.py

CELERY_BROKER_URL = 'amqp://<USER>:<PASSWORD>@chimpanzee.rmq.cloudamqp.com/<USER>'

And that's after putting the the following line in celery.py to tell it to look in settings.py for configs.

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app.config_from_object('django.conf:settings', namespace='CELERY')

Push it to heroku, and see if that solves your problem. If you're trying to start the worker on windows locally, you have to use the eventlet option in your terminal. Hope this helps, and let me know if you're still having problems.