CODEWITHSUNDEEP
ajaxnode.jsmongodbweb-crawler
You.Brighton
You.Brighton

Reputation: 2068

how to crawling it make about jquery site when I use node

I should crawl https://motul.lubricantadvisor.com/Default.aspx?data=1&lang=ENG&lang=eng

but how can I do crawl the this website. I think it use jQuery. some people say you should use ajax. but I'll contain database by mongodb so I'll use node.js how can I do that?

Upvotes: 1

Views: 647

Answers (1)

Aditya R
Aditya R

Reputation: 567

Instead of using NodeJS (designed for other purposes), use PhantomJS, which is specially designed for testing/scraping of webpages. Since it uses JavaScript, it should be pretty easy to learn for you.

Another method (if you want to use Node) is to figure out how this webpage communicates to the underlying backend and connect directly to the backend using a library such as node-XMLHttpRequest.

Yet another option is to scrape data directly from the webpage using artoo.js, which injects directly into the rendered webpage and allows you to scrape the webpage using jQuery selectors.

Ethics note: However, as with all scraping, please be careful and only scrape websites for which you have explicit permission to. Not only may you be stealing their data, you may be wasting their bandwidth (and therefore their money), so please be considerate when using any sort of scraping tool.

Upvotes: 1

Related Questions