CODEWITHSUNDEEP
phpcurldata-mining
Deepak Krishna
Deepak Krishna

Reputation: 49

Faster way to download pages using curl?

Hi,

I download a large amount of files for data mining. I used to use PHP for this purpose but I am finding it to be too slow. Also I just want a small part of the web page. I want to achieve two things

  1. Curl should be able to utilize all my download bandwidth
  2. Is there any way to download only a part of the web page where my data resides.

I am not confined to PHP. If curl works better in terminal I would use that.

Upvotes: 0

Views: 1711

Answers (3)

Daniel Stenberg
Daniel Stenberg

Reputation: 57994

Yes, you can download only a part of the page by using the CURLOPT_RANGE option, and you can also provide a write callback function that simply returns an error when you've received "enough" data and you want to stop and move on.

Upvotes: 2

Patrick Fisher
Patrick Fisher

Reputation: 8055

There is no way to download only part of a page. When you request a URL, the server response is what it is.

Utilize more of your bandwidth by using cURL's ability to make multiple connections at once.

Upvotes: 0

JohnP
JohnP

Reputation: 50009

Are you downloading HTML? Your comment leads me to believe that you are. If that's the case, simply load up the html with Simple PHP DOM and get only the part that you want. Although, I find it hard to believe that grabbing just the HTML is slowing you down. Are you downloading any files or media as well?

Link : http://simplehtmldom.sourceforge.net/

Upvotes: 0

Related Questions