CODEWITHSUNDEEP
crawler4j
Ramesh
Ramesh

Reputation: 2337

crawler4j to crawl a list of urls without crawling entire website

I have a list of web URLS need to be crawl. Is that possible to crawl only the list of webpage s with out crawling it deep. If i add the url as seed it crawls full website with full depth.

Upvotes: 1

Views: 903

Answers (1)

Hopiu
Hopiu

Reputation: 456

To only crawl the pages which you added as a seed, set the MaxDepthOfCrawling to 0.

CrawlConfig config = new CrawlConfig();
config.setMaxDepthOfCrawling(0);
PageFetcher pageFetcher = new PageFetcher(config);

Upvotes: 3

Related Questions