How to keep Laravel Queue system running on server

Question

I recently setup a Laravel Queue system. The basics are a cronjob calls a command which adds jobs to a queue and calls a second command which sends an email.

The system works when I ssh into my server and run php artisan queue:listen, but if I close my terminal the listener shuts down and the jobs stack up and sit in queue until I ssh back in and run listen again.

What is the best way to keep my queue system running in the background without needing to keep my connection open via ssh?

I tried running php artisan queue:work --daemon, and it completed the jobs in the queue, but when I closed my terminal it closed the connection and the background process.

Answer 1

Note: Laravel 10 now has 'process' interaction This should be used to overcome when your host blocks the pid lookup such as siteground

see: https://laravel.com/docs/10.x/releases#process

---

From https://gist.github.com/ivanvermeyen/b72061c5d70c61e86875

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;

class EnsureQueueListenerIsRunning extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'queue:checkup';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Ensure that the queue listener is running.';

    /**
     * Create a new command instance.
     */
    public function __construct()
    {
        parent::__construct();
    }

    /**
     * Execute the console command.
     *
     * @return void
     */
    public function handle()
    {
        if ( ! $this->isQueueListenerRunning()) {
            $this->comment('Queue listener is being started.');
            $pid = $this->startQueueListener();
            $this->saveQueueListenerPID($pid);
        }

        $this->comment('Queue listener is running.');
    }

    /**
     * Check if the queue listener is running.
     *
     * @return bool
     */
    private function isQueueListenerRunning()
    {
        if ( ! $pid = $this->getLastQueueListenerPID()) {
            return false;
        }

        $process = exec("ps -p $pid -opid=,cmd=");
        //$processIsQueueListener = str_contains($process, 'queue:listen'); // 5.1
        $processIsQueueListener = ! empty($process); // 5.6 - see comments

        return $processIsQueueListener;
    }

    /**
     * Get any existing queue listener PID.
     *
     * @return bool|string
     */
    private function getLastQueueListenerPID()
    {
        if ( ! file_exists(__DIR__ . '/queue.pid')) {
            return false;
        }

        return file_get_contents(__DIR__ . '/queue.pid');
    }

    /**
     * Save the queue listener PID to a file.
     *
     * @param $pid
     *
     * @return void
     */
    private function saveQueueListenerPID($pid)
    {
        file_put_contents(__DIR__ . '/queue.pid', $pid);
    }

    /**
     * Start the queue listener.
     *
     * @return int
     */
    private function startQueueListener()
    {
        //$command = 'php-cli ' . base_path() . '/artisan queue:listen --timeout=60 --sleep=5 --tries=3 > /dev/null & echo $!'; // 5.1
        //$command = 'php-cli ' . base_path() . '/artisan queue:work --timeout=60 --sleep=5 --tries=3 > /dev/null & echo $!'; // 5.6 - see comments

 //handle memory issues
 $command = env('PATH_PHP') . ' ' . base_path() . '/artisan queue:work --queue=default --delay=0 --memory=256 --once --timeout=60 --sleep=5 --tries=3 > /dev/null & echo $!';

        $pid = exec($command);

        return $pid;
    }
}

Answer 2

I executed with this command and my command was executed in the background and after that I took ‍htop and saw that my command was running and I trusted it, maybe it will help you.

In addition, if there is a problem, it will be recorded in the Laravel log.

nohup php artisan queue:work --daemon >> storage/logs/laravel.log 2>&1 &

Answer 3

Running

nohup php artisan queue:work --daemon &

Will prevent the command exiting when you log out.

The trailing ampersand (&) causes process start in the background, so you can continue to use the shell and do not have to wait until the script is finished.

See nohup

nohup - run a command immune to hangups, with output to a non-tty

This will output information to a file entitled nohup.out in the directory where you run the command. If you have no interest in the output, you can redirect stdout and stderr to /dev/null; similarly, you could output it into your normal Laravel log. For example

nohup php artisan queue:work --daemon > /dev/null 2>&1 &

nohup php artisan queue:work --daemon >> storage/logs/laravel.log &

But you should also use something like Supervisord to ensure that the service remains running and is restarted after crashes/failures.

Use >> to append to laravel.log. Using a single > will replace the file each time.

Answer 4

There is more than one way to skin this cat depending on your environment, and its configuration capabilities. Some are on shared hosting, some have whm access...

A more direct and maintainable way is is the following;

In your kernal file add $schedule->command('queue:work')->everyFiveMinutes()->withoutOverlapping(); to your schedule method.

This will run the queue work command every five minutes.

/**
 * Define the application's command schedule.
 *
 * @param Schedule $schedule
 * @return void
 */
protected function schedule(Schedule $schedule)
{
    $schedule->command('queue:work')->everyFiveMinutes()->withoutOverlapping();
}

See the laravel documentation - https://laravel.com/docs/9.x/scheduling#defining-schedules

Answer 5

Using AWS SQS Connection on Ubuntu

Installing Supervisor

sudo apt-get install supervisor

Configuring Supervisor

step 1 : goto /etc/supervisor/conf.d directory

cd /etc/supervisor/conf.d

step 2 : create a worker file laravel-worker.conf that will listen queue

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=/opt/bitnami/php/bin/php /opt/bitnami/nginx/html/website/queue:work sqs
autostart=true
autorestart=true
user=root
numprocs=2
redirect_stderr=true
startsecs=0
stdout_logfile=/opt/bitnami/nginx/html/website/storage/logs/supervisord.log

4)sudo supervisorctl reread

when run this command get output queue-worker:available

5)sudo supervisorctl update

when run this command get output queue-worker:added process group

other command

1)sudo supervisorctl reload

when run this command get output Restarted supervisord

2)sudo service supervisor restart

3)sudo supervisorctl status

see the status of the process

Answer 6

You can run command line:

php artisan queue:listen --queue=queue_name --tries=1 --memory=128 --timeout=300 >>  storage/logs/queue_log.log &

Check process running:

ps aux | grep php

Answer 7

I achieved the result without any service monitor or third party software. The solution is working fine but I am not sure if it is the best way.

Solution

Just run cli command in the following way in your function.

use Illuminate\Console\Command;

public function callQueue()
{
        $restart = 'php-cli ' . base_path() . '/artisan queue:restart > /dev/null & echo $!'; 
        $work = 'php-cli ' . base_path() . '/artisan queue:work --timeout=0 --sleep=5 --tries=3 > /dev/null & echo $!';
        exec($restart);
        exec($work);
}

$job = (new jobName())->delay(Carbon::now()->addSeconds(5));
dispatch($job);

Reason

The reason I have used these two commands is because the command associated with $restart prevent having any memory issue according to a comment in this answer and the command associated with $work ensures that the command is successfully executed before the job.

Answer 8

For systems with systemd as init service you could use the following service, adapting it to your project (create it on /etc/systemd/system/queue-handler.service):

[Unit]
Description = Queue Handler - Project
After = network-online.target, mysql.service

[Service]
User = www-data
Type = simple
WorkingDirectory=/var/www/project
ExecStart = /usr/bin/php /var/www/project/artisan queue:work --tries=3
Restart = on-failure
RestartSec=5s
RestartPreventExitStatus = 255

[Install]
WantedBy = multi-user.target

Reload the configurations and enable it on boot:

$ systemctl enable queue-handler.service
$ systemctl daemon-reload

Answer 9

You can also use Docker containers.

checkout:

• Docker + Laravel queue:work
• https://laravel-news.com/laravel-scheduler-queue-docker

Answer 10

For CentOS7

yum install supervisor

Then create a file in /etc/supervisord.d/filename.ini With content

[program:laravel-worker]
command=/usr/bin/php /home/appuser/public_html/artisan queue:listen
process_name=%(program_name)s_%(process_num)02d
numprocs=5
priority=999
autostart=true
autorestart=true
startsecs=1
startretries=3
user=appuser
redirect_stderr=true
stdout_logfile=/path/logpath/artisan.log

Then start the supervisord service using

systemctl restart supervisord

Enable supervisord service to run on boot using

systemctl enable supervisord

Check if the service is running using

ps aux | grep artisan

You should see the process running if it was set up properly. Similar to the output below.

[root@server ~]# ps aux | grep artisan
appuser 17444  0.1  0.8 378656 31068 ?        S    12:43   0:05 /usr/bin/php /home/appuser/public_html/artisan queue:listen

Answer 11

Installing Supervisor

sudo apt-get install supervisor

Configuring Supervisor

step 1 : goto /etc/supervisor/conf.d directory

cd /etc/supervisor/conf.d

step 2 : create a worker file laravel-worker.conf that will listen queue

sudo nano laravel-worker.conf

*Note : Now assuming that your laravel app is inside /var/www/html directory

project folder is : /var/www/html/LaravelApp

step 3 : paste below code in laravel-worker.conf and save file

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/LaravelApp/artisan queue:listen redis --queue=default --sleep=3 --tries=3 
autostart=true
autorestart=true
user=root
numprocs=8
redirect_stderr=true
stdout_logfile= /var/www/html/LaravelApp/storage/logs/worker.log

*Note : Here is assumed that you are using redis for queue connection

in .env file QUEUE_CONNECTION=redis

command=php /var/www/html/LaravelApp/artisan queue:listen redis

if you are using other connection then , general syntax will be :

command= php [project_folder_path]/artisan queue:listen [connection_name]

[connection_name] can be any of sync , database , beanstalkd , sqs , redis

step 4 : create a worker file laravel-schedule.conf that will run artisan schedule:run command at every 1 minute (60 seconds) (*you can change it as per your requirement)

[program:laravel-schedule]
process_name=%(program_name)s_%(process_num)02d
command=/bin/bash -c 'while true; do date && php /var/www/html/LaravelApp/artisan schedule:run; sleep 60; done'
autostart=true
autorestart=true
numprocs=1
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0

step 5 : Starting Supervisor : run below commands

sudo supervisorctl reread

sudo supervisorctl update

sudo supervisorctl start all

*Note : Whenever you make changes in any of configuration .conf files , run above commands of Step 5

Extra usefull information :

• to stop all supervisorctl process : sudo supervisorctl stop all
• to restart all supervisorctl process : sudo supervisorctl restart all

usefull links :

https://laravel.com/docs/5.8/queues#running-the-queue-worker

http://supervisord.org/index.html

Answer 12

I simply used php artisan queue:work --tries=3 & which keeps the process running in the background. But it sometimes stops. I don't know why this is happening

Edit

I solved this issue by using supervisor. Put a supervisor script that runs this php script, and that will run every time the server runs

Answer 13

1)sudo apt install supervisor or

sudo apt-get install supervisor

2)cd /etc/supervisor/conf.d 3)create new file inside

sudo vim queue-worker.conf

File Content

[program:email-queue]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/laravelproject/artisan queue:work
autostart=true
autorestart=true
user=root
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/html/laravelproject/storage/logs/supervisord.log

4)sudo supervisorctl reread

when run this command get output queue-worker:available

5)sudo supervisorctl update

when run this command get output queue-worker:added process group

other command

1)sudo supervisorctl reload

when run this command get output Restarted supervisord

2)sudo service supervisor restart

Answer 14

The best way is PM2 (Advanced, production process manager for Node.js) that you can monit your queues and see their's logs.

with command below in your project directory, run queue worker :

pm2 start artisan --name laravel-worker --interpreter php -- queue:work --daemon

Answer 15

Since this was a Laravel-specific question, I thought I would suggest a Lravel-specific answer. Since you are already using cronjobs on this server, I would recommend that you set up the shell command as a recurring cronjob to always verify that the worker is running. You could either set up the shell command to run natively through cron on your server, or you could use the Laravel console kernel to manage the command and add logic, such as checking whether you already have a worker running and, if not, go ahead and start it back up.

Depending on how often you need to run your command, you could do this as infrequently as once a week, or even once a minute. This would give you the ability to make sure that your workers are continuously running, without having to add any overhead to your server, such as Supervisor. Giving permissions to a 3rd party package like supervisor is ok if you trust it, but if you can avoid needing to rely on it, you may want to consider this approach instead.

An example of using this to do what you want would be to have a cronjob that runs each hour. It would execute the following in sequential order from within a custom Laravel console command:

\Artisan::call('queue:restart');

\Artisan::call('queue:work --daemon');

Note that this applies for older versions of Laravel (up to 5.3) but I haven't tested on newer versions.

Answer 16

Using pm2

I had JS script running with pm2 (Advanced, production process manager for Node.js) Which was the only one I was running. But now as I got one more process to keep running.

I created process.yml to run both with a single command. Check the first one would run php artisan queue: listen

# process.yml at /var/www/ which is root dir of the project
apps:
  # Run php artisan queue:listen to execute queue job
  - script    : 'artisan'
    name      : 'artisan-queue-listen'
    cwd       : '/var/www/'
    args      : 'queue:listen' # or queue:work
    interpreter : 'php'

  # same way add any other script if any.

Now run:

> sudo pm2 start process.yml

Check more options and feature of pm2

Answer 17

For those who are already running NodeJS on their production environments. I use PM2 to manage app processes.

# install
npm install -g pm2

# in project dir with your CI or dev setup tool 
# --name gives task a name so that you can later manage it
# -- delimits arguments that get passed to the script
pm2 start artisan --interpreter php --name queue-worker -- queue:work --daemon

I use Vagrant in development and setup NodeJS and this process using only inline vagrant scripts.

When you use PM2 in development you can use one of the many watchers to manage the restart. Simply run pm2 restart queue-worker when you pick up a change. In production I don't recommend this approach, rather opt for a build tool that can follow this process.

# 1. stop pm task to ensure that no unexpected behaviour occurs during build
pm2 stop queue-worker
# 2. do your build tasks
...
# 3. restart queue so that it loads the new code
pm2 restart queue-worker

Answer 18

You can use monit tool. it's very small and useful for any type of process management and monitoring.

After Downloading binary package from this link, you can extract it to a folder on your system and then copy two files from the package to your system to install it:

cd /path/to/monit/folder
cp ./bin/monit /usr/sbin/monit
cp ./conf/monitrc /etc/monitrc  

Now edit /etc/monitrc base on your needs(reference doc). then create a init control file to enable monit on startup. now start monit like this:

initctl reload-configuration
start monit

Answer 19

You should use linux supervisor

Installation is simple and on Ubuntu I can install it with following command:

apt-get install supervisor

Supervisor configuration files are located in /etc/supervisor/conf.d directory.

[program:email-queue]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/laravel-example/artisan queue:work redis --queue=emailqueue --sleep=3 --tries=3
autostart=true
autorestart=true
user=forge
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/laravel-example//storage/logs/supervisord.log

For each process you should create a new process configuration file. With this configuration, listener will retry each job 3 times. Also Supervisor will restart listener if it fails or if system restarts.

Answer 20

The command

nohup php artisan queue:work --daemon &

was correct, it would allow the process to continue after closing the SSH connection; however, this is only a short term fix. Once your server is rebooted or any issue causes the process to stop you will need to go back and run the command again. When that occurs, you never know. It could happen on a Friday night, so it is better to implement a long term solution.

I ended up switching over to Supervisord, this can be installed on Ubuntu as easy as

sudo apt-get install supervisor 

For AWS-AMI or RedHat users you can follow the set of instructions I outlined in this question:

Setting up Supervisord on a AWS AMI Linux Server

Answer 21

What if you start the listening within a screen? See here: http://aperiodic.net/screen/quick_reference Then even if you log out, the screen would still be active and running. Not sure why the daemonization doesnt work though.