How can a function be run multiple times at the same time in php?

Question

I have a page to run with cron. The function on this page sends a request to a link and the sample json returned from this request is below.

First Request

 {
  "next": "randomToken123",
  "previous": null,
  "items": [
    {
     ....
    },
    {
     ....
    }
   ]
 }

Second Request

     {
      "next": "randomToken456",
      "previous": null,
      "items": [
        {
         ....
        },
        {
         ....
        }
       ]
     }

The main thing I want to do here is to be able to send another request without waiting for the actions to be taken after the previous request, by getting the next value returned after the first request. So, let me show you with an example of a function:

public function cronFunc($next_cursor = null) {
    $result = doRequest($next_cursor);
    updateTheDatabaseTable($result);
}

I want to add or update the returned items to the database at the same time by running the above function 3 times at the same time. So, I want to perform 3 transactions at the same time. Is it possible to do this with threads? Or is there another way? I tried to do it using the header('Location:...') function, but after a while it gave an error saying too_many_redirects.

There will be about 1 million+ data and each request will do 200 updates. So, since I want to run it 3 times here, I will have made 600 updates at once.

Answer 1

I offer two solutions for approaching this. One is using multithreading and one is using cronjobs.

Solution using threads in PHP

There are libraries for supporting multithreading in PHP.

Install PCNTL and alfallouji/php_multithread

You can install this multithreading library with:

composer require alfallouji/php_multithread

Which depends on PCNTL which comes with the following package:

sudo apt-get install php-cli

Concept

• (1) Run first request without token
• (1) Get response with next token
• (1) Start new thread with new token
• • (2) Run second request with new token
• • (2) Get response with next token
• • (2) Start new thread with new token
• • ... third request etc....
• • (2) Process second request
• (1) Process first request

Example using alfallouji/php_multithread

This code is a simulation of the concept explained above. Add the following code to test.php, and run php test.php:

<?php

require_once __DIR__ . "/vendor/autoload.php";

$manager = new Threading\Multiple(100);

class Task extends \Threading\Task\Base
{
    private $params = [], $manager;

    public function __construct(array $params, \Threading\Multiple $manager)
    {
        $this->params = $params;
        $this->manager = $manager;
    }

    public function initialize() 
    {
        return true;
    }

    public function onSuccess()
    {
        return true;
    }

    public function onFailure() 
    {
        return false;
    }

    # For demo; get next token; or false
    private function getToken()
    {
        return array_rand([rand(1000, 99999)=>0,rand(1000, 99999)=>0,rand(1000, 99999)=>0,-1=>-1]);
    }

    public function process(array $params = array())
    {
        echo getmypid() . "(token: ".$this->params['token'] . ") - init" . PHP_EOL;
        sleep(1); # fake some heavy process
        if(!isset($this->params['token'])) {
            echo getmypid() . "(token: ".$this->params['token'] . ") - Do first request here " . PHP_EOL;
            $this->manager->start(new Task(['token'=>'randomTokenFirst'], $this->manager));
        } elseif(isset($this->params['token']) && $this->params['token'] !== -1) {
            $token = $this->getToken();
            echo getmypid() . "(token: ".$this->params['token'] . ") - New token obtained (".$token.") " . PHP_EOL;
            $this->manager->start(new Task(['token'=>$token], $this->manager));
        } else {
            echo getmypid() . "(token: ".$this->params['token'] . ") - Last request done " . PHP_EOL;
        }

        sleep(1); # fake some heavy process

        //Handle request here
        echo getmypid() . "(token: ".$this->params['token'] . ") - Processing " . PHP_EOL;

        return true;
    }
}

$manager->start(new Task(['token'=>0], $manager));

The output will be some debug information. It will virtually run from the first request (without token) to the end:

280004(token: 0) - init
280004(token: 0) - New token obtained (64865) 
280005(token: 64865) - init
280004(token: 0) - Processing 
280005(token: 64865) - New token obtained (68564) 
280006(token: 68564) - init
280005(token: 64865) - Processing 
280006(token: 68564) - New token obtained (54573) 
280007(token: 54573) - init
280006(token: 68564) - Processing 
280007(token: 54573) - New token obtained (63093) 
280008(token: 63093) - init
280007(token: 54573) - Processing 
280008(token: 63093) - New token obtained (89230) 
280010(token: 89230) - init
280008(token: 63093) - Processing 
280010(token: 89230) - New token obtained (82375) 
280011(token: 82375) - init
280010(token: 89230) - Processing 
280011(token: 82375) - New token obtained (-1) 
280012(token: -1) - init
280011(token: 82375) - Processing 
280012(token: -1) - Last request done 
280012(token: -1) - Processing 

Solution using cronjobs

I will explain a simple concept which should be tweaked further to your own needs. PHP does officially not include multithreading, so most often these solutions are implemented using cronjobs.

Using a database/ json

You could set up a database table which keeps track of the last token from the request. Or you could store the token in a json for example. This will provide the information to the next execution of the script.

Running the cronjob every x seconds

You could run a cronjob every five seconds and read the token value from the database or json:

* * * * * ( php script.php )  
* * * * * ( sleep 5 ; php script.php )  
* * * * * ( sleep 10 ; php script.php )  
* * * * * ( sleep 15 ; php script.php )  
* * * * * ( sleep 20 ; php script.php )  
* * * * * ( sleep 25 ; php script.php ) 
* * * * * ( sleep 30 ; php script.php ) 
* * * * * ( sleep 35 ; php script.php ) 
* * * * * ( sleep 40 ; php script.php ) 
* * * * * ( sleep 45 ; php script.php ) 
* * * * * ( sleep 50 ; php script.php ) 
* * * * * ( sleep 55 ; php script.php ) 

Additional logic

You could prevent the cronjob from executing the same token twice, by implementing a counter or a boolean value in the database along with the token value. And you should provide some logic to know when the end is reached (no more tokens are returned for the next request).

Switching to Java

If you want to limit the number of requests made at the same time, my advice is to switch to Java. PHP does not natively support multithreading, so if lacks a lot of features that are included in Java. For example a Java Semaphore object can limit the number of requests done, based on a number of permits, at the same time to a specific resource.