CODEWITHSUNDEEP
phpajaxcurl
Andy
Andy

Reputation: 5414

Downloading many web pages with PHP curl

I'm building a PHP application which has a database containing approximately 140 URLs.

The goal is to download a copy of the contents of these web pages.

I've already written code which reads the URL's from my database then uses curl to grab a copy of the page. It then gets everything between <body> </body>, and writes it to a file. It also takes into account redirects, e.g. if I go to a URL and the response code is 302, it will follow the appropriate link. So far so good.

This all works OK for a number of URLs (maybe 20 or so) but then my script times out due to the max_execution_time being set to 30 seconds. I don't want to override or increase this, as I feel that's a poor solution.

I've thought of two workarounds but would like to know if these are a good/bad approach, or if there are better ways.

The first approach is to use a LIMIT on the database query such that it splits the task up into 20 rows at a time (i.e. run the script 7 separate times, if there were 140 rows). I understand from this approach it still needs to call the script, download.php, 7 separate times so would need to pass in the LIMIT figures.

The second is to have a script where I pass in the ID of each individual database record I want the URL for (e.g. download.php?id=2) and then do multiple Ajax requests to them (download.php?id=2, download.php?id=3, download.php?id=4 etc). Based on $_GET['id'] it could do a query to find the URL in the database etc. In theory I'd be doing 140 separate requests as it's a 1 request per URL set up.

I've read some other posts which have pointed to queueing systems, but these are beyond my knowledge. If this is the best way then is there a particular system which is worth taking a look at?

Edit

There are 140 URLs at the moment, and this is likely to increase over time. So I'm looking for a solution that will scale without hitting any timeout limits.

Upvotes: 0

Views: 263

Answers (4)

Misunderstood
Misunderstood

Reputation: 5661

Execute all the requests simultaneously using stream_socket_client(). Save all the socket IDs in an array

Then loop through the array of IDs with stream_select() to read the responses.

It's almost like multi-tasking within PHP.

Upvotes: 0

LordNeo
LordNeo

Reputation: 1182

When you have an unknown list wich will take an unknown amount of time asynchronous calls are the way to go.

Split your script into a single page download (like you proposed, download.php?id=X).

From the "main" script get the list from the database, iterate over it and send an ajax call to the script for each one. As all the calls will be fired all at once, check for your bandwidth and CPU time. You could break it into "X active task" using the success callback.

You can either set the download.php file to return success data or to save it to a database with the id of the website and the result of the call. I recommend the later because you can then just leave the main script and grab the results at a later time.

You can't increase the time limit indefinitively and can't wait indefinitively time to complete the request, so you need a "fire and forget" and that's what asynchronous call does best.

As @apokryfos pointed out, depending on the timing of this sort of "backups" you could fit this into a task scheduler (like chron). If you call it "on demand", put it in a gui, if you call it "every x time" put a chron task pointing the main script, it will do the same.

Upvotes: 1

George Dimitriadis
George Dimitriadis

Reputation: 1815

What you are describing sounds like a job for the console. The browser is for the users to see, but your task is something that the programmer will run, so use the console. Or schedule the file to run with a cron-job or anything similar that is handled by the developer.

Upvotes: 0

Thomas Nikoleris
Thomas Nikoleris

Reputation: 172

I dont agree with your logic , if the script is running OK and it needs more time to finish, just give it more time it is not a poor solution.What you are suggesting makes things more complicated and will not scale well if your urls increase.

I would suggest moving your script to the command line where there is no time limit and not using the browser to execute it.

Upvotes: 2

Related Questions