Celery [Errno 13] Permission denied: '/var/run/celery/beat.pid'

Question

I've been running celery on server for a while and today when I tried to restart celery by using

sudo systemctl restart celery 

The process somehow failed. I did journalctl -xe and this is the error message that i got:

This error might have occured when I reboot my ubuntu server.

Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:   File "/home/baefloral/baefloral/fandeedv3/lib/python3.7/site-packages/celery/apps/beat.py", line 77, in run
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:     self.start_scheduler()
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:   File "/home/baefloral/baefloral/fandeedv3/lib/python3.7/site-packages/celery/apps/beat.py", line 88, in start_s
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:     platforms.create_pidlock(self.pidfile)
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:   File "/home/baefloral/baefloral/fandeedv3/lib/python3.7/site-packages/celery/platforms.py", line 260, in create
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:     pidlock = _create_pidlock(pidfile)
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:   File "/home/baefloral/baefloral/fandeedv3/lib/python3.7/site-packages/celery/platforms.py", line 270, in _creat
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:     pidlock.acquire()
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:   File "/home/baefloral/baefloral/fandeedv3/lib/python3.7/site-packages/celery/platforms.py", line 151, in acquir
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:     reraise(LockFailed, LockFailed(str(exc)), sys.exc_info()[2])
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:   File "/home/baefloral/baefloral/fandeedv3/lib/python3.7/site-packages/celery/exceptions.py", line 108, in rerai
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:     raise value.with_traceback(tb)
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:   File "/home/baefloral/baefloral/fandeedv3/lib/python3.7/site-packages/celery/platforms.py", line 149, in acquir
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:     self.write_pid()
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:   File "/home/baefloral/baefloral/fandeedv3/lib/python3.7/site-packages/celery/platforms.py", line 218, in write_
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Aug 09 15:19:01 OdinAsk-nyc1-Ubuntu-01 sh[14937]: celery.platforms.LockFailed: [Errno 13] Permission denied: '/var/run/celery/beat.pid'

Here is my configuration file

# Name of nodes to start
# here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"


# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/baefloral/baefloral/fandeedv3/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="Reddit_site"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=2700 --concurrency=8"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
#   and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"

# you may wish to add these options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"

Before this error, I was facing ERO20: [Errno 20] Not a directory: '/var/run/celery/beat.pid' so, I've checked /var/run/celery and notice that celery is not a directlyon /var/run for some reasons, but it is rather an empty file. I remove the file and made it into an empty directory

I've tried some suggestions from previous questions, I tried to type and didnt work

sudo chmod 755 /var/log/celery/ /var/run/celery/
sudo chown root:root /var/log/celery/ /var/run/celery/

I am running the tasks on a ubuntu remote server on digitalocean and using a virtual environment and not a root user to run it

Answer 1

I ran into similar situation you had. Here's the fix I had on the Ubuntu server. Since Ubuntu rebuild /var/run on every reboot, the config below will automatically create directories needed by apps on every reboot.

Add a file called celery.conf in /etc/tmpfiles.d.

In this file, add one line:

d /var/run/celery 0755 <user> <group>

Replace user and group with what is configured on your system for celery systemd instance.