CODEWITHSUNDEEP
phphttppostweb-scrapingfile-get-contents
Rob
Rob

Reputation: 275

file_get_contents and variables

I have a text file filled with a list of IDs. Using PHP, I am trying to load a url for each of the IDs and pull something from that page (another id)

For example, if I have the IDs 555, 888, 222 I want to load the URLs

http://example.edu/bvl.P_Sel?facultyID=555

http://example.edu/bvl.P_Sel?facultyID=888

http://example.edu/bvl.P_Sel?facultyID=222

I tried to get the content via
file_get_contents("http://example.edu/bvl.P_Sel?facultyID=$lines[0]");

where $lines is an array of the IDs. This returns the following error:

Warning: file_get_contents(http://example.edu/bvl.P_Sel?facultyID=222)    [function.file-get-contents]: failed to open stream: HTTP request failed! HTTP/1.1 404

That url is an example, but the url in the error does work when I visit it manually. And if I replace the file_get_contents variable with an actual ID, like ?facultyID=222, it works perfectly.

I visited this question's answer How to post data in PHP using file_get_contents? and tried assigning a variable in the $postdata array to a variable, and I get the same exact error only with ?facultyID=XXX removed from the error message's url.

My implementation of the latter is here.

Upvotes: 0

Views: 1479

Answers (3)

Crsr
Crsr

Reputation: 624

$lines = array(813667,1124279,760643,668461,2868,33613);
print_r($lines);

output:

Array ( [0] => 813667 [1] => 1124279 [2] => 760643 [3] => 668461 [4] => 2868 [5] => 33613 )

this:

foreach($lines as $key => $value):
echo '<pre>';
print_r($lines[$key]);
endforeach;

output:

    813667
1124279
760643
668461
2868
33613

and this: $get = file_get_contents("http://example.edu/bvl.P_Sel?facultyID=$lines[0]"); print_r($get);

output:

  Example Domains
  As described in RFC 2606,
    we maintain a number of domains such as EXAMPLE.COM and EXAMPLE.ORG
    for documentation purposes. These domains may be used as illustrative
    examples in documents without prior coordination with us. They are 
    not available for registration.

What is wrong? :) it's what you need?

Upvotes: 1

Eduardo Reveles
Eduardo Reveles

Reputation: 2175

Those encoded characters when you use the urlencode function (%0D%0A) are a new line, so maybe you array of id's have them in each element. Try this:

// your code to generate the lines array
file_get_contents("http://example.edu/bvl.P_Sel?facultyID=" . trim($lines[0]));

Upvotes: 2

Hmayak Tigranyan
Hmayak Tigranyan

Reputation: 62

Try to use CURL for scraping and also post data as it is more powerful and more advanced.

Upvotes: 0

Related Questions