CODEWITHSUNDEEP
apacheweb-crawlernutch
tt2244
tt2244

Reputation: 71

Nutch not crawling entire website

I am using nutch 2.3.1

I preform the commands to crawl a site:

The problem is, nutch is only crawling the first URL (the one specified in seeds.txt). The data is only the HTML from the first URL/page.

All the other URLS that were accumulated by the generate command are not actually crawled.

I cannot get nutch to crawl the other generated urls...I also cannot get nutch to crawl the entire website. What are the options that I need to use to crawl an entire site?

Does anyone have any insights or recommendations?

Thank you so much for your help

Upvotes: 2

Views: 1299

Answers (1)

Do Do
Do Do

Reputation: 723

In the case that Nutch crawls only one specified URL, please check Nutch filter (conf/regex-urlfilter.txt). To crawl all URLs in the seed, the content of regex-urlfilter.txt should be as follows.

# accept all URLs
+.

See details here: http://wiki.apache.org/nutch/NutchTutorial

Hope this helps,

Le Quoc Do

Upvotes: 1

Related Questions