CODEWITHSUNDEEP
phplaravel
user1436631
user1436631

Reputation:

Laravel Queues: How to process one set of jobs only after another set of jobs are processed?

I have two sets of jobs:

How can I process all the jobs in Set1 and then process the jobs in Set2?

I tried job chaining. But job chaining stops processing the jobs if its previous job is failed. I want to process them all even if one of them is failed.

It should process all the jobs in Set1(A,B,C), and the process of B and C should not stop if the process of job A failed. When all the jobs of Set1 are processed(no matter failed or succeeded) It should pick up jobs in Set2 to process.

The order of the jobs is not priority here. They can be processed in any order.
The only rule is that jobs in Set2 should only be processed after all the jobs in Set1 are processed.

I cannot work with delayed dispatch because the time taken to process Set1 of jobs varies very highly.

Upvotes: 0

Views: 5747

Answers (4)

Mouagip
Mouagip

Reputation: 5319

This can now be done with Job Batching in Laravel 8.x:

$batch = Bus::batch([
    new ImportCsv(1, 100),
    new ImportCsv(101, 200),
    new ImportCsv(201, 300),
    new ImportCsv(301, 400),
    new ImportCsv(401, 500),
])->then(function (Batch $batch) {
    // All jobs completed successfully...
})->catch(function (Batch $batch, Throwable $e) {
    // First batch job failure detected...
})->finally(function (Batch $batch) {
    // The batch has finished executing...
})->dispatch();

Upvotes: 1

Travis Britz
Travis Britz

Reputation: 5552

You can dispatch them to different queues:

A::dispatch()->onQueue('set1');
B::dispatch()->onQueue('set1');
C::dispatch()->onQueue('set1');

D::dispatch()->onQueue('set2');
E::dispatch()->onQueue('set2');
F::dispatch()->onQueue('set2');

and run a worker that makes sure set1 is empty before checking set2:

php artisan queue:work --queue=set1,set2

To start a worker that verifies that all of the high queue jobs are processed before continuing to any jobs on the low queue, pass a comma-delimited list of queue names to the work command:

php artisan queue:work --queue=high,low

https://laravel.com/docs/5.8/queues#running-the-queue-worker

Edit:

You can solve set1 jobs from User2 delaying the set2 jobs from User1 by starting more workers.

Upvotes: -1

Marcus
Marcus

Reputation: 1848

An idea would be to create a JobSet job and a JobWorker job.

So the Job set takes your new JobWorker("A"), JobWorker("B"), JobWorker("C") as an argument and then dispatches all of them. 

dispatch(new JobSet(new JobWorker("A"), new JobWorker("B"), new JobWorker("C")));

and the "JobSet" job for inspiration

class JobSet implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    private $a;
    private $b;
    private $c;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct(JobWorker $a, JobWorker $b, JobWorker $c)
    {
        $this->a = $a;
        $this->b = $b;
        $this->c = $c;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        dispatch($this->a);
        dispatch($this->b);
        dispatch($this->c);
    }
}

Upvotes: 0

Mihir Bhende
Mihir Bhende

Reputation: 9045

The job chaining in Laravel is based on the fact that it will no execute the next jobs if current one fails.

You can use events. Each job has 2 methods handle() and failed(). You can emit events either when job is executed successfully or it is failed. See below example :

For example : 

<?php

namespace App\Jobs;

use Exception;
use App\Something;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;

class JobA implements ShouldQueue
{
    use InteractsWithQueue, Queueable, SerializesModels;

    protected $something;

    /**
     * Create a new job instance.
     *
     * @param  Something $somethin
     * @return void
     */
    public function __construct(Something $something)
    {
        $this->something = $something;
    }

    /**
     * Execute the job.
     *
     * @param  AudioProcessor  $processor
     * @return void
     */
    public function handle()
    {
        // Execute Job on something
        event(new JobAHandled($something));
    }

    /**
     * The job failed to process.
     *
     * @param  Exception  $exception
     * @return void
     */
    public function failed(Exception $exception)
    {
        // Send user notification of failure, etc...
        event(new JobAFailed($something, $e));
    }
}

Now, you can call the next job by listening to events JobAHandled and JobAFailed. This will give you control that even if you are continuing the next job execution, you know what the earlier job failed. You can log that if needed.

Then you can do the same for other jobs independent of set it is in.

Also make sure if you try the job, then the further jobs will be executed again as considering your case the failed event will execute the next job as well.

Upvotes: 0

Related Questions