André Gollubits
André Gollubits

Reputation: 193

Laravel symfony process exceeded the timeout of 60 seconds

I have been having an issue in Laravel which I just can't seem to solve. As a part of our application, we are adding a lot of jobs to a queue and executing them using queue:work. This is setup with Supervisor and is working very well.

We are having trouble with just this one specific command (the rest are run without problems).

In our application, we are calling a custom shell script command using

Symfony\Component\Process\Process;

But I keep getting the "process exceeded the timeout of 60 seconds" error, no matter what.

Error: Symfony\Component\Process\Exception\ProcessTimedOutException: The process "'BIMTool.sh'" exceeded the timeout of 60 seconds. in /var/www/app/vendor/symfony/process/Process.php:1263

The BimTool.sh script iniates a series of commands involving Python & Anaconda3 to generate BIM-model data from IFC and point clouds. I even replaced the contents of BimTool.sh with "sleep 70" and I still got the error when run.

This is the code in question:

GenerateModelData.php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Support\Facades\Log;
use Symfony\Component\Process\Exception\ProcessFailedException;
use Symfony\Component\Process\Process;
use Carbon\Carbon;

use App\Models\MongoDB\VisualizerModel;
use App\Models\PointCloud;
use App\Models\Project;

class GenerateModelData implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

protected $pointcloud;

/**
 * The number of times the job may be attempted.
 *
 * @var int
 */
public $tries = 3;

 /**
 * The number of seconds the job can run before timing out.
 *
 * @var int
 */
public $timeout = 1200;

/**
 * The number of seconds to wait before retrying the job.
 *
 * @var int
 */
public $backoff = 120;

/**
 * Create a new job instance.
 *
 * @return void
 */
public function __construct(PointCloud $pointcloud)
{
    $this->pointcloud = $pointcloud;
}

/**
 * Execute the job.
 *
 * @return void
 */
public function handle()
{
    // These commands can take +8 hours to execute in some cases
    ini_set('max_execution_time', 0);
    set_time_limit(0);

    $pointcloud = $this->pointcloud;

    $modelID = $pointcloud->models[0]->model_id;

    $model = VisualizerModel::find($modelID);

    $IFCFileName = substr($model->filename, 0, -5);

    $IFCFileLocation = "/var/www/app/storage/app/public/application/projects/{$model->project_uuid}/models/ifc/{$IFCFileName}.ifc";
    $PTSFileLocation = "/var/www/app/storage/app/public/{$pointcloud->storage_location}";
    $downSample = 15;
    $uuid = $model->project_uuid;
    $date = Carbon::parse($pointcloud->scan_date)->format('d/m/Y');
    $modelDataID = $model->model_data_id;

    $process = new Process(['BIMTool.sh',
        $IFCFileLocation,
        $PTSFileLocation,
        $downSample,
        $uuid,
        $modelID,
        $date,
        $modelDataID 
    ]);

    $process->setTimeout(1200);
    $process->setIdleTimeout(1200);
    $process->start();

    // Executes after the command finishes
    if (!$process->isSuccessful()) {

        throw new ProcessFailedException($process);

    } else {
        // Command was succesfull

        // Some code here

    }
}

/**
 * Catch error if job fails.
 *
 * @return \Symfony\Component\Process\Exception\ProcessFailedException
 */
public function failed(ProcessFailedException $exception)
{
   return $exception;
}

I have of course searched Stackoverflow (and the internet for that matter) for this problem already and have tried all the solutions I came across, but with no effect.

What I have tried so far (and none of them has worked)

Environment configuration

Linode running Ubuntu 18.04.5 LTS
LAMP-stack
Anaconda3
Laravel 5.8

Supervisor configuration

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work database --sleep=5 --tries=3 --timeout=0 -- 
queue=default
autostart=true
autorestart=true
user=bc
numprocs=1
redirect_stderr=true
stdout_logfile=/var/www/app/worker.log

What could be causing the process exceeded the timeout of 60 seconds error? I keep getting it no matter what I attempt.

EDIT: (this solved it for me)

So it turned out the queue has a "cache" as well. Running the command:

php artisan queue:restart

cleared the queue of cached versions of the jobs. What went wrong was that my changes to the code was never run, so make sure to run this command in order to make sure the job queue is using the most recent code.

Thx for the inputs!

Upvotes: 5

Views: 11700

Answers (1)

Yevgeniy Afanasyev
Yevgeniy Afanasyev

Reputation: 41430

Error: Symfony\Component\Process\Exception\ProcessTimedOutException: The process "'BIMTool.sh'" exceeded the timeout of 60 seconds. in /var/www/app/vendor/symfony/process/Process.php:1263

This error can only come from Symfony\Component\Process\Process; and it is easily fixable by

$process->setTimeout(3600);

more in documentation

https://symfony.com/doc/current/components/process.html

But you are executing it in a queue. So, most likely when you update your PHP code it does not get updated in laravel queue cache. Especially if you running your queues in production mode using supervisor. Which is not recommended for development. However if you do it this way then, make sure you run

php artisan queue:restart 

every time you make a change in PHP code.

Or better for a development environment use

php artisan queue:work

once the queue:work command has started, it will continue to run until it is manually stopped or you close your terminal

more in documentation

https://laravel.com/docs/9.x/queues#the-queue-work-command

Upvotes: 4

Related Questions