CODEWITHSUNDEEP
phpffmpegqueue
Ben
Ben

Reputation: 301

PHP queue implementation

I'm new to using queues for intensive processes. I have an app that will take an upload of a short video, process it with FFMPEG, then upload to youtube using their API, then interact with my database. My questions:

Should I use two different queues? One to process, then hand it off to a different queue to upload? Or should I place all my processing in one worker?

Is it ok to interact with a database from a worker or should I do this some other way?

Upvotes: 1

Views: 1527

Answers (1)

Maksim Kotlyar
Maksim Kotlyar

Reputation: 3937

I would suggest using two different queues, one for image processing and one for uploading it to youtube. That's perfectly okay to query a database from queue workers, though you may pass all the required data in the message so you do not need a db.

Here's how you can implement something like this using enqueue library.

You have to install the enqueue/simple-client library and one of the transports. Assuming you chose the filesystem one, so let's install it:

composer require enqueue/simple-client enqueue/fs

Now let's see how you can send messages from your POST script:

<?php
// producer.php

use Enqueue\SimpleClient\SimpleClient;

include __DIR__.'/vendor/autoload.php';

$client = new SimpleClient('file://'); // the queue will store messages in tmp folder

// you uploaded the file to your server,
// and now you have a path to the file.
// I assume it is in the $movieFile var.

$client->sendCommand('process_movie', $movieFile);

The consumption script: 

<?php
// consumer.php

use Enqueue\Client\Config;
use Enqueue\SimpleClient\SimpleClient;
use Enqueue\Psr\PsrProcessor;
use Enqueue\Psr\PsrMessage;

include __DIR__.'/vendor/autoload.php';

$client = new SimpleClient('file://');

$client->bind(Config::COMMAND_TOPIC, 'process_movie',  function(PsrMessage $psrMessage) use ($client) {
   $movieFile = $psrMessage->getBody();

   // a movie processing logic here

   // once we have done with processing we can send a message to upload_movie queue.

   $client->sendCommand('upload_movie', $movieFile);

   return PsrProcessor::ACK;
});

$client->bind(Config::COMMAND_TOPIC, 'upload_movie', function(PsrMessage $psrMessage) {
   $movieFile = $psrMessage->getBody();

   // a movie uploading logic here

   return PsrProcessor::ACK;
});

// this call is optional but it worth to mention it.
// it configures a broker, for example it can create queues and excanges on RabbitMQ side. 
$client->setupBroker();

$client->consume();

Run as many consumer.php processes as you by using supervisord or other process managers, on local computer you can run it without any extra libs or packages.

That's a basic example and enqueue has a lot of other features that might come in handy. If you are interested, check the enqueue documentation out.

Upvotes: 1

Related Questions