CODEWITHSUNDEEP
javascriptc#web-crawler
qubits
qubits

Reputation: 1307

Spider crawling framework in JS

This is not going to be a strict tech programming question. I am looking more for an advice, and hope you guys can help.

I am looking for a web framework/spider that could crawl an irregular site (a site that does not have an easily definable page structure, for example using title-strings in url to index) pointed to by me and search for keywords. Something that can be easily customized and where I can easily program additional functionalities in JS (preferably) or C#

I am green in the web crawling industry, so just throw me a framework, and il be happy to read it up myself. Thanks :)

Upvotes: 0

Views: 180

Answers (1)

jimbo
jimbo

Reputation: 582

For me the best C# web crawler is still Abot https://code.google.com/p/abot/ It's easy to learn and can be extended.

Upvotes: 1

Related Questions