How to execute a long task in background?

Question

I have an app made using Symfony 5, and I have a script that upload a video located on the server to the logged-in user channel.

Here's basically the code of my controller:

    /**
     * Upload a video to YouTube.
     *
     * @Route("/upload_youtube/{id}", name="api_admin_video_upload_youtube", methods={"POST"}, requirements={"id" = "\d+"})
     */
    public function upload_youtube(int $id, Request $request, VideoRepository $repository, \Google_Client $googleClient): JsonResponse
    {
        $video = $repository->find($id);

        if (!$video) {
            return $this->json([], Response::HTTP_NOT_FOUND);
        }

        $data = json_decode(
            $request->getContent(),
            true
        );

        $googleClient->setRedirectUri($_SERVER['CLIENT_URL'] . '/admin/videos/youtube');
        $googleClient->fetchAccessTokenWithAuthCode($data['code']);

        $videoPath = $this->getParameter('videos_directory') . '/' . $video->getFilename();

        $service = new \Google_Service_YouTube($googleClient);
        $ytVideo = new \Google_Service_YouTube_Video();

        $ytVideoSnippet = new \Google_Service_YouTube_VideoSnippet();
        $ytVideoSnippet->setTitle($video->getTitle());
        $ytVideo->setSnippet($ytVideoSnippet);

        $ytVideoStatus = new \Google_Service_YouTube_VideoStatus();
        $ytVideoStatus->setPrivacyStatus('private');
        $ytVideo->setStatus($ytVideoStatus);

        $chunkSizeBytes = 1 * 1024 * 1024;

        $googleClient->setDefer(true);

        $insertRequest = $service->videos->insert(
            'snippet,status',
            $ytVideo
        );

        $media = new \Google_Http_MediaFileUpload($googleClient, $insertRequest, 'video/*', null, true, $chunkSizeBytes);
        $media->setFileSize(filesize($videoPath));

        $uploadStatus = false;
        $handle = fopen($videoPath, "rb");

        while (!$uploadStatus && !feof($handle)) {
            $chunk = fread($handle, $chunkSizeBytes);
            $uploadStatus = $media->nextChunk($chunk);
        }

        fclose($handle);
    }

This basically works, but the problem is that the video can be very big (10G+), so it's taking a very long time, and basically Nginx terminates before it's ended and returns a "504 Gateway Timeout" before the upload is completed.

And anyway, I don't want the user to have to wait for a page to load while it's uploading.

So, I'm looking for a way to, instead of just immediately running that script, execute that script in some kind of background thread, or in a asynchronous way.

The controller returns a 200 to the user, I can tell him that uploading is happening and to come back later to check progress.

How to do this?

Answer 1

There are many ways to accomplish this, but what you basically want is to decouple the action trigger and its execution.

Simply:

• Remove all heavy work from your controller. Your controller should at most just check that the video id provided by the client exists in your VideoRepository.

• Exists? Good, then you need to store this "work order" somewhere.

  There are many solutions for this, depending on what you have already installed, what technology you feel more comfortable with, etc.

  For sake of simplicity, let's say you have a PendingUploads table, with videoId, status, createdAt and maybe userId. So the only thing your controller would do is to create a new record in this table (maybe checking that the job is not "queued" yet, that kind of detail is up to your implementation).

• And then return 200 (or 202, which could be more appropriate in the circumstances)

You would need then to write a separate process.

Very likely a console command that you would execute regularly (using cron would be the simplest way)

On each execution that process (which would have all the Google_Client logic, and probably a PendingUploadsRepository) would check which jobs are pending to upload, process them sequentially, and set status to whatever you signify to done. You could have status to either 0 (pending), 1 (processing), and 2 (processed), for example, and set the status accordingly on each step of the script.

The details on exactly to implement this are up to you. That question would be too broad and opinionated. Pick something that you already understand and allows you to move faster. If you are storing your jobs in Rabbit, Redis, a database, or a flat-file is not particularly important. If you start your "consumer" with cron or supervisor, either.

Symfony has a ready made component that could allow you to decouple this kind of messaging asynchronously (Symfony Messenger), and it's pretty nice. Investigate if it's your cup of tea, although if you are not going to use it for anything else in your application I would keep it simple to begin with.