CODEWITHSUNDEEP
node.jsweb-scrapinguser-agent
David D.
David D.

Reputation: 567

Setting a User Agent in scrape-it

I'm using scrape-it in my node.js scraping tool (for identifying proper keyword usage) but being identified as a bot by some websites and not getting any content. Is there a way to configure a known user agent header for the GET request to bypass the block?

Upvotes: 3

Views: 513

Answers (1)

Douglas Reid
Douglas Reid

Reputation: 3788

You can set the headers, including User-agent, by passing an options object to scrape-it:

scrapeIt({
    url: "http://example.com"
  , headers: { "User-agent": "known-user-agent-of-choice" }
},
{
  // some scrapeHTML options ...
})
.then(
 // some code ...
);

Upvotes: 3

Related Questions