CODEWITHSUNDEEP
php
Marty Wallace
Marty Wallace

Reputation: 35734

long running php script that loads a large amount of data

I have a PHP script will be ran on a production web server only once. The script loads a whole lot of db records and then iterates over them, json_decoding a field and checking it. Some fields will be amended and saved back to the database.

The script will consume a lot of memory and will potentially run for 1hr or more.

What should I keep in mind with this type of script with regards to making sure it doesn’t cause the site to go down or any other potential hazards of such scripts.

The basic flow of the script is as follows:

1. load all records from db table
2. iterate over each row
  2.1 json_decode specific field
  2.2 if a matching field is found then
      2.2.1 Change the value 
      2.2.2 Save back to the database

Upvotes: 0

Views: 277

Answers (3)

Ben
Ben

Reputation: 104

Some important DB factors to consider.

I would do all the updates in a DB transaction outside of stage 2 at the end, instead of the loop.

You should look at your DB engine, see if it uses row / table locking. This could certainly stop data from being read/written if its being used elsewhere.

I don't know your data set, but below are some ideas which could help increase the script speed...

Using isset() instead of array_key_exists() for array keys is a lot faster, watch out for NULL values though.

Maybe hashing keys from DB and comparing it against a hash of your new data set. Only need to check against one value instead of multiple values in each row. Could save a lot of iterations or key checks.

Upvotes: 1

75inchpianist
75inchpianist

Reputation: 4102

Make sure you set it up using a Cron job, or the like, and run it at low load times. This varies depending on the users of your site, so you'll have to analyze your visits to determine what a good window would be.

If this only needs to be run once, it might make sense to put up a maintenance window notice for visitors. This will allow them to plan ahead to not expect a super responsive site between the given hours.

Upvotes: 3

Lix
Lix

Reputation: 47956

If you are dealing with a large amount of data and some heavy processing and your main concern is to maintain the availability of the server for other tasks (such as serving a website), you might want to consider splitting up the task into smaller chunks and having a cron job process each job periodically.

Other than that you should definitely take a look at the set_time_limit() function. With it you can ensure that your script will not timeout by passing a value of zero - 

set_time_limit(0);

Upvotes: 0

Related Questions