CODEWITHSUNDEEP
apacheweb-crawlernutch
Hafiz Muhammad Shafiq
Hafiz Muhammad Shafiq

Reputation: 8670

How to allow apache nutch to crawl forever

I am using apache nutch (2.2.1) for crawling. What changes are need if I want to crawl forever. Guide me completely as I am not familiar with nutch too mutch.

Upvotes: 1

Views: 145

Answers (1)

Do Do
Do Do

Reputation: 723

If you want to crawl forever, the following is the script you need:

#!/bin/bash

./bin/nutch inject urls #urls is the seed data
while [ 1 == 1 ]
do
   ./bin/nutch generate -topN 10000 # 10000 is the number of URLs will be fetch in each crawling round, you can modify it
   ./bin/nutch fetch -all
   ./bin/nutch parse -all
   ./bin/nutch updatedb

done

Hope this helps

Le Quoc Do

Upvotes: 2

Related Questions