Retrieve urls that are related using web crawler

Question

I am using opensearchserver to crawl a website. Now I want to get all related urls for a specific url. The idea is when someone reads an article on a website they get a related articles option at the bottom of the page. how can I do this using opensearchserver? is this even possible? if not is there any other web crawler that can do such a thing?

Thanks

Answer 1

For beginning sorry for this response very late. For this you have in the crawler, the field mapping tab but before you have to configure the shema tab. the tab name Parser list, lists every available parser. The field mapping, with this you can fill all fields you want.