CODEWITHSUNDEEP
php
David Sword
David Sword

Reputation: 113

Cron a PHP loop that requires page refresh

I have a PHP crawler that fires from crontab a few times a day

php crawler.php

In crawler.php, I query a database for my sites, then loop and check them: 

foreach ($mysql_results as .. ) {
    // curl and check html for my checks
    // each site takes about 30sec
}

I've begun changing the logic, to instead just do one site per php execution, finish, then re-fire the script and run the next site

// get offset from database
// based on offset get single site from database
// curl and check html for my checks
// increase offset by one
// close database
// run this page again until no more sites are left
shell_exec('php '.__FILE__);

Everything fine until, of course, that last line. shell_exec waits for the execution to finish.. If I'm not mistaken, my php file is not firing independently as a sibling process - it's just stacking within itself, which is what I'm trying to get away from.

Is there anyway in php, to fire a php cli process - without waiting? Or is my logic just totally off?

Upvotes: 1

Views: 186

Answers (2)

Mladen Ilić
Mladen Ilić

Reputation: 1775

You are right, it's executes the script synchronously.

Instead, better approach would be to have cron run a script once a minute and than have a script update "last crawl date" for a particular website in the database.

That way each new script run should just pick up the site with the oldest "last crawl date" and process it.

This way, each script run would be completely independent and you could have multiple php processes running in parallel.

Cheers.

Upvotes: 1

Barmar
Barmar

Reputation: 782130

you can use pcntl_exec(), this is the PHP equivalent of the POSIX execve() system call. It replaces the current process with the new program.

pcntl_exec("/usr/bin/php", ["php", __FILE__]);

Upvotes: 1

Related Questions