CODEWITHSUNDEEP
phpcronweb-crawler
user13094663
user13094663

Reputation:

Schedule Thousands Of Tasks [PHP Scripts] On a Server

Let's say a website who needs to display the updated content every 5 minutes. For sure we can use a cron job to schedule a PHP script like

$weather = file_get_contents("https://weather.com/country/state/city/day/hour/minute.json");
$upload_to_database = send_data_to_db();

Let's say this simple script takes the data and send it to the MYSQL and then the data is being displayed on frontend by fetching data using AJAX every few minutes or maybe sending the notifications after analyzing the data and so on...

Now, what if we have thousands of cities. How can I create cron jobs for those cities automatically?

I hope you did understand my question.

Another example can be SERP Rank tracker for thousands of keywords and so on...

Upvotes: 0

Views: 109

Answers (1)

RocketNuts
RocketNuts

Reputation: 11130

Instead of creating many separate cron jobs which each do one retrieve/update job, just create one generic cron job which does all the retrieval/updating work.

If you want them to run independently and simultaneously, you could spawn separate processes from that one cron job. You could do that dynamically from PHP, so you can use a current list of cities and spawn a separate process for each.

For example, instead of running "php updatecity.php Washington" (as an example of how you run your php script that updates a particular city), run this:

nohup php updatecity.php "$city" > /dev/null 2>&1 &

This will launch a separate php process silently in the background, running your updatecity.php script with a $city parameter as argument.

Make sure your processes cannot stall or keep running, otherwise you may end up flooding your server with tons of unterminated processes.

Upvotes: 2

Related Questions