CODEWITHSUNDEEP
php
Bogdan
Bogdan

Reputation: 461

Searching within a webpage

what would be the best way to write a code in Php that would search within a webpage for a number of words stored in a file? is it best to store the source code in a file or is it another way? please help.

Upvotes: 0

Views: 547

Answers (1)

The Surrican
The Surrican

Reputation: 29856

The best way is to use google: site:example.com word1 OR word2 OR word3

Do you want to search in ONE PAGE? or one website with MULTIPLE PAGES? If its only one page i think you can store the html code in memory without problems. if you know exactly what you search strpos for reach word will probably be the fastest (stripos for case insensitive). you can also define your own character class and use preg_match_all or something... just something like this will do...

<?
$keywords = array("word1","word2","word3");
$doc = strip_tags(file_get_contents("http://www.example.com")); // remove tags to get only text
$doc = preg_replace('/\s+/', ' ',$doc); // remove multiple whitespaces...
foreach($keywords as $word) {
  $pos = stripos($doc,$word);
  if($pos !== false) {
    echo "match: ...".str_replace($word,"<em>$word</em>",substr($doc,$pos-20,50))."... \n";
  }
}
?>

something like the following for example will perform MUCH faster as its based on hashmap lookups with O(1) and doesnt need to scan the whole text for every keyword...

<?
setlocale(LC_ALL, "en_US.utf8");
$keywords = array("word1","word2","word3","word4");
$doc = file_get_contents("http://www.example.com");
$doc = strtolower($doc);
$doc = preg_replace('!/\*.*?\*/!s', '', $doc);
$doc = preg_replace("/<!--.*>/i", "", $doc);
$doc = preg_replace('!<script.*?script>!s', '', $doc);
$doc = preg_replace('!<style.*?style>!s', '', $doc);
$doc = strip_tags($doc);
$doc = preg_replace('/[^0-9a-z\s]/','',$doc);
$doc = iconv('UTF-8', 'ASCII//TRANSLIT', $doc); // check if encoding is really utf8
//$doc = preg_replace('{(.)\1+}','$1',$doc); remove duplicate chars ... possible step to add even more fuzzyness
$doc = preg_split("/\s+/",trim($doc));
foreach($keywords as $word) {
  $word = strtolower($word); 
  $word = iconv('UTF-8', 'ASCII//TRANSLIT', $word);
  $key = array_search($word,$doc);
  var_dump($key);
  if($key !== false) {
    echo "match: ";
    for($i=$key;$i<=5 && isset($doc[$i]);$i++) {
      echo $doc[$i]." ";
    }
  }
} 
?>

this code is untested.

it would be however be more elegant to dump textnodes from a domdocument

Simple searching is easy. If you want to search in a whole website the crawling logic is difficult.

I once did a backlink-checker for a company that worked like a crawler.

My first advice is not to do a recursion (like scanning a page and following all links and following all links in that until you reach a certain level...)

rather do it like this:

  • do a for loop as often as many levels you want to crawl.

  • set a site array with one entry (start page)

  • pass array to a function downloads every link, scans the site there and stores links on it in array. when done with all links return the new link list array

  • in the for loop update the array with the return value of the function, and call the function again.

this way you can avoid following nasty paths but rather crawl website level by level. also store already visited links in an array to skip, dont follow external links, check for weird url parameters etc..

for future use you can store documents in lucene or solr, there are classes to turn html pages into senseful lucene objects and search within.

Upvotes: 1

Related Questions