CODEWITHSUNDEEP
php
Augustoandro
Augustoandro

Reputation: 11

Web crawler in PHP not able to create records in mysql database. How to fix?

I am creating a web crawler for a search engine like Google. The web crawler works well, that's what I see when I run it through terminal, but it is not writing any records in the mysql database.

I have already tried granting all permissions to the database user the web crawler uses, but it was of no use. My server is flawless, that I can be sure of.

<?php
$start = "http://localhost/mariophp/test.html";
$already_crawled=array();
$crawling=array();
function get_details($url)
{
    $options=array('http'=>array('method'=>"GET", 'headers'=>"User-Agent: ZeroBot/0.2\n"));
    $context=stream_context_create($options);
    $doc = new DOMDocument();
    @$doc->loadHTML(@file_get_contents($url,false,$context));
    $title=$doc->getElementsByTagName("title");
    $title=$title->item(0)->nodeValue;
    $simg=$doc->getElementsByTagName("img");
    //$simg=$simg->getAttribute("src");
    //$simg=$simg->item(0)->nodeValue;
    $description="";
    $keywords="";
    $metas=$doc->getElementsByTagName("meta");
    for($i=0; $i<$metas->length; $i++)
    {
        $meta=$metas->item($i);
        if($meta->getAttribute("name")==strtolower("description"))
            $description=$meta->getAttribute("content");
        if($meta->getAttribute("name")==strtolower("keywords"))
            $keywords=$meta->getAttribute("content");
    }
    $_con=mysqli_connect("localhost","augustus","password");
    mysqli_select_db($_con,"websited");

    $title=$_POST["title"];
    $url=$_POST["url"];
    $keywords=$_POST["keywords"];
    $description=$_POST["description"];
    $simg=$_POST["simg"];

    $sql="insert into websited(stitle,slink,skey,sdesc,simg) values('$title','$url',$keywords',$description','$simg')"; 
    if(!mysqli_query($_con,$sql))
       {
        echo "Error: mysqli_error($_con))";
       }       

}
function follow_links($url)
{
    global $already_crawled;
    global $crawling;
    $options=array('http'=>array('method'=>"GET", 'headers'=>"User-Agent: MarioBot/0.1\n"));
    $context=stream_context_create($options);
    $doc = new DOMDocument();
    @$doc->loadHTML(@file_get_contents($url,false,$context));
    $linklist = $doc->getElementsByTagName("a");
    foreach ($linklist as $link)
    {
        $l = $link->getAttribute("href");
        if(substr($l,0,1)=="/" && substr($l,0,2)!="//")
        {
            $l=parse_url($url)["scheme"]."://".parse_url($url)["host"].$l;
        }
        else if (substr($l,0,2)=="//") 
        {
            $l=parse_url($url)["scheme"].":".$l;
        }
        else if(substr($l,0,2)=="./")
        {
            $l=parse_url($url)["scheme"]."://".parse_url($url)["host"].dirname(parse_url($url)["path"]).substr($l,1);
        }
        else if(substr($l,0,1)=="#")
        {
            $l=parse_url($url)["scheme"]."://".parse_url($url)["host"].parse_url($url)["path"].$l;
        }
        else if(substr($l,0,3)=="../")
        {
            $l=parse_url($url)["scheme"]."://".parse_url($url)["host"]."/".$l;
        }
        else if(substr($l,0,11)=="javascript:")
        {
            continue;
        }
        else if(substr($l,0,5)!="https" && substr($l,0,4)!="http")
        {
            $l=parse_url($url)["scheme"]."://".parse_url($url)["host"]."/".$l;
        }
        if(!in_array($l,$already_crawled))
        {
            $already_crawled[]=$l;
            $crawling[]=$l;
            echo get_details($l)."\n";
            //echo $l."\n";
        }
    }
    array_shift($crawling);
    foreach ($crawling as $site) {
        follow_links($site);
    }
}
follow_links($start);
print_r($already_crawled);
?>

Note:The test.html file mentioned in the code is a simple file containing links to different websites. You may need to setup a server first to successfully run this code. I am getting the following output right now. [augustoandro@Augustus zerophp]$ php crawle2.php PHP Notice: Undefined index: title in /srv/http/zerophp/crawle2.php on line 30 PHP Notice: Undefined index: url in /srv/http/zerophp/crawle2.php on line 31 PHP Notice: Undefined index: keywords in /srv/http/zerophp/crawle2.php on line 32 PHP Notice: Undefined index: description in /srv/http/zerophp/crawle2.php on line 33 PHP Notice: Undefined index: simg in /srv/http/zerophp/crawle2.php on line 34 PHP Recoverable fatal error: Object of class mysqli could not be converted to string in /srv/http/zerophp/crawle2.php on line 39 [augustoandro@Augustoandro zerophp]$

Please help.

Upvotes: 0

Views: 143

Answers (1)

Barmar
Barmar

Reputation: 782315

Get rid of these lines:

    $title=$_POST["title"];
    $url=$_POST["url"];
    $keywords=$_POST["keywords"];
    $description=$_POST["description"];
    $simg=$_POST["simg"];

These are overwriting the variables that you got from scraping the website. $_POST is for getting parameters that are submitted from a form or AJAX, they're not needed here.

The call to mysqli_error() should not be inside a string. Change

    if(!mysqli_query($_con,$sql))
       {
        echo "Error: mysqli_error($_con))";
       }

to

    if(!mysqli_query($_con,$sql))
       {
        echo "Error: " . mysqli_error($_con));
       }

Upvotes: 1

Related Questions