CODEWITHSUNDEEP
phpmysqlcurlinsertscreen-scraping
Rob B
Rob B

Reputation: 21

Inserting scraped data using php curl into MySQL

I have been working on this script for the last couple of days and cannot seem to find a way to insert the data into MySQL. I am a beginner when it comes to PHP/MYSQL and have only written a couple of simple scripts before. I am able to echo out the scraped data and get no error messages, but when I check phpmyadmin the query isn't working (the results aren't being input to the database).

Here is the code that I have been working on

require ("mysqli_connect.php"); 
include('../simple_html_dom.php');

ini_set('user_agent', 
  'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3');

// get DOM from URL or file
$html = file_get_html('http://www.asos.com/Women/Jeans/Cat/pgecategory.aspx?cid=3630&via=lhn');
// find all images

foreach($html->find('#items') as $a)       
    echo $a->innertext .'<br>';

foreach($html->find('span.price') as $p)
    echo $p->innertext .',';    

$q = "INSERT INTO jeans (`image`, `price`) VALUES ('$a', '$p')";
$r = @mysqli_query ($dbc, $q) or die ("Update query failed : " . mysql_error());;  //Run the Query.

Upvotes: 0

Views: 3101

Answers (4)

user1444442
user1444442

Reputation: 109

Remember to use mysql_real_escape_string function to avoid troubles saving your contents. Example below.

$query2 = sprintf("INSERT INTO jeans(image, price)  
                 VALUES ('%s','%s')",mysql_real_escape_string($a),
                                     mysql_real_escape_string($p))
mysql_query($query2) or die(mysql_error()."<br />".$query2);

Upvotes: 0

edwardmp
edwardmp

Reputation: 6601

You are suppressing the error by putting an @ before mysqli_query

Try changing this:

$q = "INSERT INTO jeans ('image', 'price') VALUES ('$a', '$p')";
mysqli_query ($dbc, $q) or die ("Update query failed : " . mysql_error());;  //Run the Query.

Upvotes: 1

edwardmp
edwardmp

Reputation: 6601

I' think I know what is wrong.

If you just use

echo file_get_contents("http://www.asos.com/Women/Jeans/Cat/pgecategory.aspx?cid=3630");

(or just asos.com, doesn't matter), you will notice that you'll always get back:

Sorry, we can't find that page or something has gone wrong...

go back or try here:

women men home help desk

Because none of the elements you specified are on this page, you're code will return nothing.

My guess is that asos.com prevents scraping from their site.

Upvotes: 0

somedev
somedev

Reputation: 1053

In your sample code, $a and $p are objects, try this instead:

$a = '';
foreach($html->find('#items') as $item) {
    $a.= $item->innertext .'<br>';
}

$p = '';
foreach($html->find('span.price') as $price) {
    $p.= $price->innertext.',';    
}

Next, remove the @ from @mysqli_query, try not to use that, ever, try to catch/handle errors properly instead.

Next, please take a few minutes to research paramaterized queries and PDO, don't accept unknown input (from 3rd parties no less) and inject them right in to your sql:

$q = "INSERT INTO jeans (`image`, `price`) VALUES ('$a', '$p')";

ie: Don't do that ^

Finally, you probably want to validate the response from the get.

Hope that helps!

Upvotes: 2

Related Questions