CODEWITHSUNDEEP
web-crawlernutch
grm
grm

Reputation: 5407

Getting nutch to prioritize frequently updated pages?

Is there a way to get Nutch to increase the crawling of pages that gets updated frequently?

E.g. index pages and feeds.

It would also be of value to refresh fresh pages that contains comments more frequently the first date after the page was created. Any tips are appreciated.

Upvotes: 1

Views: 322

Answers (1)

Pascal Dimassimo
Pascal Dimassimo

Reputation: 6928

What you need is the Adaptive Fetch Schedule. I have written a blog post about how it works. Basically what this scheduler does is gradually makes the pages that change more often to be visited more and more regularly.

Upvotes: 1

Related Questions